MS  MANAGEMENT  COLLEG 


595 

ill 


TEST  AND  EVALUATION 
MANAGEMENT  GUIDE 


AUGUST  1993 
SECOND  EDITION 


OTIC 


93  lO  2SJ94 


1 .  Munitions  testing  in  environmental  chambers  at  CSTA 

2.  Hybrid  laboratory  simulation  anechoic  chamber 
(in-the-loop  simulator  and  EW  system  test) 

3.  Data-reduction  charts 

4.  AN/MPS-39  multiple-object  tracking  radar 

5.  Position  attitude  diagram 

6.  Testing  of  training  device  that  simulates  system 
operation  under  battlefield  conditions 

7.  Mobile  data  telemetry  receiver  at  CSTA 

8.  Acoustical  vibration  test  (in  a  5,000-cubic-foot 
reverberation  chamber)  on  an  MX  missile  stage 

9.  A  3,000-psig,  flow  interrupt,  pressure  impulse,  and 
water-hammer  test  on  a  series  of  nuclear  valves 

10.  EOTS-F  Cinethrodolite  with  automatic  infrared  tracking 
system 


TEST  AND  EVALUATION 
MANAGEMENT  GUIDE 


AUGUST  1993 
SECOND  EDITION 


PUBLISHED  BY  THE 

DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE  PRESS 
FORT  BELVOIR,  VA  22060-5426 


fjyric  QUALITY  nJSPECTBD  8 


For  sale  by  the  U.S.  Govcmmenl  Printing  Office 
Superinlendcnl  of  Documents.  Mail  Slop:  SSOP.  Washington.  DC  20402-9328 

ISBN  0-16-041892-5 


ACKNOWLEDGMENTS 


I  would  like  to  acknowledge  the  valuable  contributions,  reviews,  comments  and  technical  assis¬ 
tance  from  LTC  Charles  K.  Gailey,  USA;  Lt  Col  Dennis  Van  Liere,  USAF;  LTC  Larry  Damman,  USA; 
and  Mr.  Paul  AJfieri  of  the  Test  and  Evaluation  Department  faculty.  The  cover  and  graphics  were 
developed  with  the  assistance  of  Mr.  Greg  Caruth  and  Mr.  John  Kerpchar;  the  editing  was  done  by 
Mrs.  Kay  Sondheimer  and  Mrs.  Esther  Farria.  Special  thanks  to  Mrs.  Peggy  Claxton  for  her  patient 
typing,  retyping  and  editorial  assistance. 


FOREWORD 


This  book  is  one  of  a  family  of  educational  guides  written  from  a  Department  of  Defense 
perspective;  i.e.,  non-Service  peculiar.  They  are  intended  primarily  for  use  in  the  courses  at  the 
Defense  Systems  Management  College  (DSMC)  and  secondarily  as  a  desk  reference  for  program 
and  project  management  personnel.  These  guidebooks  are  written  for  current  and  potential 
Department  of  Defense  (DOD)  acquisition  management  personnel  who  are  familiar  with  basic 
terms  and  definitions  employed  in  program  offices.  They  are  designed  to  assist  government  tmd 
industry  persormel  in  executing  their  management  responsibilities  relative  to  the  acquisition  and 
support  of  defense  systems.  They  include: 

a.  Integrated  Logistic  Support  (ILS)  Guide  (May  1986) 

b.  Mission  Critical  Computer  Resources  Management  Guide 

c.  Systems  Engineering  Management  Guide  (January  1990) 

d.  Department  of  Defense  Manufacturing  Management  Handbook  for  Program 
Managers  (April  1989). 

The  Defense  Systems  Management  College  is  the  controlling  agency  for  this  book.  Comments  and 
recommendations  relating  to  the  text  are  solicited. 

The  introduction  offers  a  perspective  of  test  and  evaluation  (T &E)  management  activities  during  the 
system  life  cycle.  Subsequent  material  in  this  book  provides  a  guide  for  managing  specific  T&E 
events.  The  past  several  decades  have  seen  the  rise  of  large,  highly-interactive  defense  systems  that 
are  often  on  the  forward  edge  of  technology.  These  systems  have  a  natural  evolutionary  process, 
or  life  cycle,  during  which  actions  taken  or  avoided  in  early  stages  can  mean  the  difference  between 
success  or  failure. 

The  T &E  process  for  DOD  materiel  acquisitions  is  a  complex  exercise  of  integrating  the  engineering 
development,  data  collection,  and  evaluations  necessary  to  satisfy  the  decision-maker's  informa¬ 
tion  requirements  on  system  performance.  Poorly  managed  T&E  does  not  support  an  informed 
decision  process  and  can  generate  schedule  slips  or  adverse  media  exposure,  leading  to  intensive 
interest  in  a  program's  status  by  the  Office  of  the  Secretary  of  Defense  (OSD)  and/or  the  Congress. 

The  objective  of  a  well-managed  T&E  program  is  to  provide  timely  and  accurate  information.  This 
guide  has  been  developed  to  assist  the  acquisition  community  in  obtaining  a  better  understanding 
of  who  the  decision-makers  are  and  determining  how  and  when  to  plan  test  and  evaluation  events. 


John  Claxton 
Professor 

Test  and  Evaluation  Department 
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I 

MODULE 

Management  of 
Test  and  Evaluation 


Test  and  Evaluation  is  a  management  tool 
and  an  integral  part  of  the  development 
process.  This  module  will  address  the  policy 
structure  and  oversight  mechanisms  in 
place  for  test  and  evaluation. 


1 

IMPORTANCE  OF 
TEST  AND  EVALUATION 


1.1  INTRODUCTION 

The  test  and  evaluation  (T&E)  process  is  an 
integral  part  of  the  systems  engineering 
process  which  identifies  levels  of  perfor¬ 
mance  and  assists  the  developer  in  correct¬ 
ing  deficiencies.  It  is  also  becoming  a  sig¬ 
nificant  element  in  the  decision-making 
process,  providing  data  supportive  of  trade¬ 
off  analysis,  risk  reduction  and  require¬ 
ments  refinement.  Programmatic  decisions 
on  system  performance  maturity  and  readi¬ 
ness  to  advance  to  the  next  phase  of  devel¬ 
opment  are  becoming  more  dependent  on 
demonstrated  performance.  The  ultimate 
customer,  the  Service-member  user,  is  con¬ 
cerned  about  neither  unit  cost  nor  produc¬ 
tion  schedule.  The  issue  of  paramount  im¬ 
portance  is  system  performance;  i.e.,  will  it 
fulfill  the  mission.  The  test  and  evaluation 
process  provides  data  to  tell  the  user  how 
well  the  system  is  performing  during  de- 
elopment  and  if  it  is  ready  for  fielding. 
The  program  manager  must  balance  the 
risks  of  cost,  schedule  and  performance  to 
keep  the  program  on  track  to  production 
and  fielding.  The  responsibility  of  deci¬ 
sion-making  authorities  centers  on  assess¬ 
ing  risk  trade-offs.  This  chapter  describes 
how  test  and  evaluation  functions  as  a  risk 
management  tool.  It  also  addresses  the  con¬ 
tribution  T&E  makes  by  providing  empiri¬ 
cal  data  before  each  milestone  review. 

1.2  TESTING  AS  A  RISK 
MANAGEMENT  TOOL 

Correcting  defects  in  weapons  has  been 
estimated  to  add  from  10-30  percent  to  the 


cost  of  each  item  (Reference  107).  Such 
costly  redesign  and  modification  efforts 
can  be  reduced  if  carefully  planned  and 
executed  test  and  evaluation  programs  are 
used  to  detect  and  fix  system  deficiencies 
sufficiently  early  in  the  acquisition  process 
(Figure  1-1).  Fixes  instituted  during  the 
Demonstration  /  Validation  (DEM  /  V  AL) 
Phase  cost  significantly  less  than  those 
required  in  the  Engineering  and  Manufac¬ 
turing  Development  (EMD)  Phase  after 
most  design  decisions  have  been  made. 

In  1983,  the  Assistant  Secretary  of  Defense 
made  the  following  statement  to  the  Senate 
Committee  on  Governmental  Affairs  re¬ 
garding  the  importance  of  T&E; 

.  .  .  the  criterion  should  not  be  how 
quickly  we  can  field  any  new  weapon, 
but  rather  how  quickly  we  can  field  a 
new  weapon  that  works.  The  only 
weapons  that  would  be  significantly 
delayed  would  be  the  ones  that  opera¬ 
tional  testing  shows  to  be  unsuitable 
for  combat,  and  I  cannot  believe  that 
any  of  us  would  advocate  saddling 
our  fighting  forces  with  any  of  those. 

In  fact,  the  most  likely  effect  of  opera¬ 
tional  testing  is  not  to  delay,  but  to 
accelerate  the  development  process. 
Trying  to  fix  a  faulty  weapon  after  it's 
in  the  field  —  if  it  can  still  be  fixed — is 
a  far  slower  process  than  fixing  the 
design  before  it  goes  into  production. 
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Figure  1-1.  System  Life-Cycle  Technical  Activities 


Thus,  T&E  may  reduce  cost,  schedule  and 
technical  risks.  A  third  type  of  risk  involved 
in  the  development  and  acquisition  of  new 
systems  is  technical  risk.  Test  and  evalua¬ 
tion  of  parts,  components,  subsystems  and 
systems  can  also  be  used  to  estimate  and 
manage  this  technical  risk. 

Test  and  evaluation  results  figure  promi¬ 
nently  in  the  decisions  reached  at  design 
and  milestone  reviews.  However,  the  fact 
that  T&E  results  are  required  at  major  deci¬ 
sion  points  does  not  presuppose  that  T&E 
results  must  always  be  favorable.  The  final 
decision  responsibility  lies  with  the  execu¬ 
tive  who  must  examine  the  critical  issues 
and  weigh  the  facts.  Only  he  can  determine 
the  weight  and  importance  that  is  to  be 
attributed  to  a  system's  diverse  capabilities 
and  shortcomings  and  the  degree  of  risk  he 
is  willing  to  accept.  The  decision-making 
authority  will  be  unable  to  make  this  judg¬ 
ment  without  a  solid  base  of  information 
provided  by  T&E.  Figure  1-2  illustrates  the 
life-cycle  cost  of  the  system  and  how  deci¬ 
sions  impact  program  expenditures. 

A  Defense  Science  Board  1983  Task  Force 
focused  on  the  reduction  of  risk  in  program 
acquisition  (Reference  42) .  This  group  made 
the  following  observations; 

•  A  poorly-designed  product  cannot  be 
properly  tested  or  produced; 

•  Control  techniques  needed  to  success¬ 
fully  complete  the  design,  test  and  produc¬ 
tion  of  an  item  dictate  the  management 
system  required; 

•  The  industrial  process  of  weapon  sys¬ 
tem  acquisition  demands  a  better  un¬ 
derstanding  and  implementation  of  basic 
engineering  and  manufactuing  disciplines; 

•  The  industrial  process  is  focused  on  the 
design,  test  and  production  of  a  product; 


•  The  design,  test  and  production  pro¬ 
cesses  are  a  continuum  of  interdependent 
disciplines.  Failure  to  perform  well  in  one 
area  will  result  in  failure  to  do  well  in  all 
areas.  When  this  happens,  as  it  does  too 
often,  a  high-risk  program  results  with 
equipment  fielded  later  and  at  far  greater 
cost  than  planned. 

The  Task  Force  developed  a  set  of  tem¬ 
plates  for  use  in  establishing  and  maintain¬ 
ing  low-risk  programs.  Each  template  de¬ 
scribes  an  area  of  risk  and  then  specifies 
technical  methods  for  reducing  that  risk. 
Program  managers  and  test  managers  may 
wish  to  consult  these  templates  for  guid¬ 
ance  in  reducing  the  risks  frequently  asso¬ 
ciated  with  test  programs.  Sample  risk 
management  templates  are  found  in  DOD 
4245.7-M,  "Transition  from  Development 
to  Production." 

1.3  THE  T&E  CONTRIBUTION  AT 
MAJOR  MILESTONES 

Test  and  evaluation  progress  is  monitored 
by  the  Office  of  the  Secretary  of  Defense 
(OSD)  throughout  the  acquisition  process. 
Their  oversight  extends  to  the  major  mate¬ 
riel  acquisitions  or  designated  acquisitions, 
which  are  about  five  percent  of  all  the  ac¬ 
quisitions  being  managed  within  DOD.  Test 
and  evaluation  officials  within  OSD  render 
independent  assessments  to  the  Defense 
Acquisition  Board,  the  Defense  Acquisi¬ 
tion  Executive,  and  the  Secretary  of  De¬ 
fense  at  each  major  system  milestone.  These 
assessments  are  based  on  the  following 
T&E  information; 

•  The  Test  and  Evaluation  Master  Plan 
(TEMP)  and  more  detailed  supporting 
documents  developed  by  responsible  Ser¬ 
vice  activities; 

•  Service  test  agency  reports  and  briefings; 
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•  Development  test  and  evaluation  data 
from  sources  such  as  Service  program  man¬ 
agers,  laboratories,  industry  developers, 
studies  and  analyses. 

At  Milestone  I,  the  OSD  T&E  assessment 
reflects  an  evaluation  of  system  concepts 
and  alternatives  based  on  specific  objec¬ 
tives  and  thresholds  established  in  an  ap¬ 
proved  preliminary  TEMP.  At  Milestone  II, 
it  includes  an  assessment  of  previously  es¬ 
tablished  test  plans  and  results.  At  Mile¬ 
stone  III,  reviews  verify  the  operational 
effectiveness  and  suitability  of  major 
weapon  systems. 

A  primary  contribution  made  by  T&E  is  the 
detection  and  reporting  of  deficiencies  that 
may  adversely  impact  the  performance 
capability  or  availability /supportability  of 
a  system.  A  deficiency  reporting  process  is 
used  throughout  the  acquisition  process  to 
report,  evaluate  and  track  system  deficien¬ 
cies  and  to  provide  the  impetus  for  correc¬ 
tive  actions. 

1.3.1  Test  Contributions  Prior 
to  Milestone  I 

During  the  Concept  Exploration  and  Defi¬ 
nition  Phase  prior  to  Milestone  I,  labora¬ 
tory  testing,  modeling  and  simulations  are 
conducted  by  the  contractor  and  the  devel¬ 
opment  agency  to  demonstrate  and  assess 
the  capabilities  of  key  subsystems  and  com¬ 
ponents.  The  test  and  simulation  designs 
are  based  on  the  requirements  documented 
in  the  Mission  Need  Statement.  Studies, 
analyses,  simulation  and  test  data  are  used 
by  the  development  agency  to  explore  and 
evaluate  alternative  concept  designs  pro¬ 
posed  to  satisfy  the  requirements.  Also 
during  this  period,  the  operational  test 
agency  (OTA)  monitors  concept  explora¬ 
tion  T&E  to  gather  information  for  future 
T&E  planning  and  to  provide  effectiveness 
and  suitability  input  desired  by  the  pro¬ 


gram  manager.  The  OTA  also  conducts 
operational  assessments,  as  feasible,  to  as¬ 
sess  the  operational  impact  of  candidate 
technical  approaches  and  to  assist  in  select¬ 
ing  preferred  alternative  systems  concepts. 

Toward  the  end  of  the  phase,  the  develop¬ 
ment  agency  prepares  the  Development 
Test  and  Evaluation  (DT&E)  System  Con¬ 
cept  Report.  This  report  records  and  pre¬ 
sents  T&E  results  of  system  design(s)  engi¬ 
neering  and  performance  evaluations  com¬ 
pared  to  stated  requirements  and  concept 
specifications.  This  information  is  incorpo¬ 
rated  into  the  Program  Manager's  Status 
Briefing  and  the  Integrated  Program  Sum¬ 
mary,  key  documents  that  form  the  basis 
for  the  Milestone  I  decision  to  proceed  to 
the  DEM/VAL  Phase. 

1.3.2  Test  Contributions  Prior  to 
Milestone  II 

During  the  DEM/VAL  Phase  prior  to  Mile¬ 
stone  II,  concepts  approved  for  demonstra¬ 
tion  and  validation  form  the  baseline  that  is 
used  for  detailed  test  planning. 

The  development  agency  conducts  devel¬ 
opment  test  and  evaluation  during  the 
DEM/VAL  Phase  to  assist  with  engineer¬ 
ing  design,  system  development  and  to 
verify  attainment  of  technical  performance 
specifications  and  program  objectives.  The 
DT&E  includes  T&E  of  components,  sub¬ 
systems  and  prototype  development  mod¬ 
els.  Test  and  evaluation  of  functional  com¬ 
patibility  and  interoperability  with  exist¬ 
ing  and  planned  equipment  and  systems  is 
also  included.  During  this  phase  of  testing, 
adequate  DT&E  is  accomplished  to  ensure 
engineering  is  reasonably  complete  (includ¬ 
ing  survivability /vulnerability,  compatibil¬ 
ity,  transportability,  interoperability,  reli¬ 
ability,  maintainability,  safety,  human  fac¬ 
tors,  and  logistic  supportability).  Also,  this 
phase  confirms  that  all  significant  design 
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problems  have  been  identified  and  solu¬ 
tions  to  these  problems  are  in  hand. 

The  Service  operational  test  and  evaluation 
agency  (OT&E)  conducts  early  operational 
assessments  to  estimate  the  system's  op¬ 
erational  effectiveness  and  suitability;  iden¬ 
tifies  needed  modifications;  and  provides 
information  on  tactics,  doctrine,  organiza¬ 
tion  and  personnel  requirements.  The 
OT&E  program  is  accomplished  in  an  envi¬ 
ronment  as  operationally  realistic  as  pos¬ 
sible.  Typical  operational  and  support  per¬ 
sonnel  are  used  to  obtain  a  valid  estimate  of 
the  user's  capability  to  operate  and  main¬ 
tain  the  system.  The  user  of  the  system 
monitors  T&E  during  the  DEM/ VAL  Phase. 
Some  of  the  most  important  products  of 
user  monitoring  are  the  attainment  of  early 
orientation  ana  advanced  training,  dem¬ 
onstrations  of  system  performance,  and 
valid  operational  testing  (OT)  assessments 
of  system  maintainability  and  supportabil- 
ity. 

The  development  agency  prepares  a  report 
on  the  results  of  demonstration  and  valida¬ 
tion  DT&E  for  review  by  the  Service  head¬ 
quarters  and  the  Service  acquisition  review 
coimcil  prior  to  system  acquisition  review 
by  DOD.  The  report  includes  the  results  of 
testing  and  supporting  information,  con¬ 
clusions  and  recommendations  for  full-scale 
development.  At  the  same  time,  the  OT&E 
agency  prepares  independent  early  opera¬ 
tional  assessments,  which  contain  estimates 
of  the  system's  operational  effectiveness 
and  suitability.  The  OT&E  assessments  pro¬ 
vide  a  permanent  record  of  OT&E  accom¬ 
plished,  an  audit  trail  of  OT&E  data,  test 
results,  conclusions  and  recommendations. 
This  information  is  used  to  support  the 
development  of  the  Integrated  Program 
Summary,  which  is  prepared  for  Milestone 
II,  and  recommends  which  of  the  alterna¬ 
tive  systems  studied  in  the  DEM/VAL 
Phase  will  proceed  into  engineering  and 
manufacturing  development. 


1.3.3  Test  Contributions  Prior  to 
Milestone  III 

Prior  to  Milestone  III,  the  objective  of  the 
EMD  Phase  is  to  design,  fabricate  and  test 
a  preproduction  system  that  closely  ap¬ 
proximates  the  final  product.  Test  and 
evaluation  activities  during  this  period  yield 
much  useful  information.  For  example,  data 
obtained  during  EMD  test  and  evaluation 
is  used  to  assist  in  evaluating  the  system's 
maintenance  training  requirements  and  the 
proposed  training  program.  Test  results 
generated  during  EMD  test  and  evaluation 
also  support  the  user  in  refining  and  updat¬ 
ing  tactics. 

During  the  EMD  Phase,  development  test 
and  evaluation  is  conducted  to  satisfy  the 
following  objectives: 

(1)  As  specified  in  program  documents, 
assess  the  critical  technical  issues: 

(a)  Determine  how  well  the  develop¬ 
ment  contract  specifications  have  been  met; 

(b)  Identify  system  technical  deficien¬ 
cies  and  appropriate  corrective  actions; 

(c)  Determine  whether  the  system  is 
compatible  and  interoperable  with  exist¬ 
ing  and  planned  equipment  or  systems; 

(d)  Estimate  the  reliability,  maintain¬ 
ability  and  availability  of  the  system  after  it 
is  deployed; 

(e)  Determine  whether  the  system  is 
safe  and  ready  for  OT&E; 

(f)  Validate  any  configuration  changes 
caused  by  correcting  deficiencies,  modifi¬ 
cations  or  product  improvements; 

(g)  Assess  human  factors  and  iden¬ 
tify  limiting  factors; 
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(2)  Assess  the  technical  risk  and  evaluate 
the  trade-offs  among  specifications,  opera¬ 
tional  requirements,  life-cycle  costs  and 
schedules; 

(3)  Assess  the  survivability,  vulnerabil¬ 
ity  and  logistic  supportability  of  the  sys¬ 
tem; 

(4)  Verify  the  accuracy  and  complete¬ 
ness  of  the  technical  documentation  devel¬ 
oped  to  maintain  and  operate  the  weapons 
system; 

(5)  Gather  information  for  training  pro¬ 
programs  and  technical  training  materi¬ 
als  needed  to  support  the  weapon  system; 

(6)  Provide  information  on  environmen¬ 
tal  issues  for  use  in  preparing  environmen¬ 
tal  impact  assessments; 

(7)  Determine  system  performance  limi¬ 
tations  and  safe  operating  parameters; 

(8)  Using  Live  Fire  Test  (LFT),  evaluate 
vulnerability  or  lethality  of  a  weapon  sys¬ 
tem  as  appropriate  and  as  required  by  law. 

Initial  OT&E  is  conducted  prior  to  the  pro¬ 
duction  decision  at  Milestone  III  to: 

(1)  Estimate  the  operational  effective¬ 
ness  and  suitability  of  the  system; 

(2)  Identify  operational  deficiencies; 

(3)  Recommend  and  evaluate  changes  in 
production  configuration; 

(4)  Provide  information  for  developing 
and  refining  logistics  support  requirements 
for  the  system  and  training,  tactics,  tech¬ 
niques  and  doctrine; 

(5)  Provide  information  to  refine  opera¬ 
tion  and  support  (O&S)  cost  estimates  and 


identify  system  characteristics  or  deficien¬ 
cies  that  can  significantly  impact  O&S  costs; 

(6)  Determine  whether  the  technical  pub¬ 
lications  and  support  equipment  are  ad¬ 
equate;  in  the  operational  environment. 

Thus,  T&E  activities  intensify  during  the 
EMD  Phase  and  make  significant  contribu¬ 
tions  to  the  overall  acquisition  decision 
process. 

1.3.4  Test  Contributions  After 
The  Production  Decision 

After  Milestone  III,  when  the  production 
decision  is  made,  T&E  activities  continue 
to  provide  important  insights.  Tests  de¬ 
scribed  in  the  TEMP  and  not  completed 
during  the  EMD  Phase  are  completed  dur¬ 
ing  the  Production  and  Deployment  Phase. 
The  residual  DT&E  is  usually  limited  to  all- 
weather  testing,  correction  of  deficiencies 
and  engineering  modifications.  System  el¬ 
ements  are  integrated  into  the  final  opera¬ 
tional  configuration,  and  development  test¬ 
ing  is  completed  when  the  system  perfor¬ 
mance  requirements  are  met.  During  the 
Production  Phase,  government  represen¬ 
tatives  normally  monitor  or  conduct  the 
production  acceptance  test  and  evaluation 
(PAT&E).  Each  system  is  verified  by  PAT&E 
for  compliance  with  the  requirements  and 
specificaHons  of  the  contract. 

Postproduction  testing  requirements  may 
result  from  an  acquisition  strategy  calling 
for  block  changes  to  accommodate  engi¬ 
neering  changes  or  the  use  of  preplanned 
product  improvements  (P^I).  This  will  al¬ 
low  parallel  development  of  high-risk  tech¬ 
nology  and  modular  insertion  of  system 
upgrades  into  production  equipment.  Tech¬ 
nology  breakthroughs  and  significant  threat 
changes  may  require  system  modifications. 
The  development  of  the  modifications  will 
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require  development  testing;  and,  if  sys¬ 
tem  performance  is  significantly  changed, 
operational  testing  may  be  appropriate. 

Operational  T&E  activities  continue  after 
the  production  decision  in  the  form  of  fol- 
low-on  operational  test  and  evaluation 
(FOT&E).  The  initial  phase  of  FOT&E  may 
be  conducted  by  either  the  OT&E  agency  or 
user  commands,  depending  on  Service  di¬ 
rectives.  It  verifies  the  operational  effec¬ 
tiveness  and  suitability  of  the  production 
system  and  determines  if  deficiencies  iden¬ 
tified  during  the  initial  OT&E  have  been 
corrected.  A  second  phase  of  FOT&E  is 
conducted  by  the  user  to  refine  doctrine, 
tactics,  techniques  and  training  programs 
for  the  life  of  the  system. 

The  OT&E  agency  prepares  a  final  report  at 
the  conclusion  of  its  management  phase  of 
FOT&E.  This  report  records  test  results, 
describes  the  evaluation  accomplished  to 
satisfy  critical  issues  and  objectives  estab¬ 
lished  for  FOT&E  and  documents  its  as¬ 
sessment  of  deficiencies  resolved  during 
EMD.  Deficiencies  that  are  not  corrected 
are  recorded  with  recommended  correc¬ 
tive  actions. 

A  final  report  on  FOT&E  is  also  prepared 
by  the  using  command  test  team  with  em¬ 
phasis  on  the  operational  utility  of  the  sys¬ 
tem  when  operated,  maintained  and  sup¬ 
ported  by  operational  personnel  using  the 
concepts  specified  for  the  system.  Specific 
attention  is  devoted  to  the  following; 

(1)  The  degree  to  which  the  system  ac¬ 
complishes  the  mission  when  employed  by 
operational  persormel  in  a  realistic  scenario 
with  the  appropriate  organization,  doctrine, 
threat  (including  countermeasures  and 
nuclear  threats),  environment  and  using 
tactics  and  techniques  developed  during 
earlier  FOT&E; 


(2)  The  degree  to  which  the  system  can 
be  placed  in  operational  field  use,  with 
specific  evaluations  of  availability,  com¬ 
patibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintain¬ 
ability,  safety,  human  factors,  manpower 
supportability,  logistics  supportability  and 
training  requirements; 

(3)  The  conditions  under  which  the  sys¬ 
tem  was  tested  including  the  natural 
weather  and  climatic  conditions,  terrain 
effects,  battlefield  disturbances  and  enemy 
threat  conditions; 

(4)  The  ability  of  the  system  to  perform 
its  required  functions  for  the  duration  of  a 
specified  mission  profile; 

(5)  System  weaknesses  such  as  the  vul¬ 
nerability  of  the  system  to  exploitation  by 
countermeasures  techniques  and  the  prac¬ 
ticality  and  probability  of  an  adversary 
exploiting  the  susceptibility  of  a  system  in 
combat. 

A  specific  evaluation  of  the  manpower  and 
logistics  changes  needed  for  the  effective 
integration  of  the  system  into  the  user's 
inventory  is  also  made.  These  assessments 
provide  essential  input  for  the  later  phases 
of  the  system  acquisition  cycle. 

1.4  SUMMARY 

"Risk  management  is  the  means  by  which 
the  program  areas  of  vulnerability  and  con¬ 
cern  are  identified  and  managed."  (Refer¬ 
ence  20).  Test  and  evaluation  is  the  disci¬ 
pline  that  helps  to  illuminate  those  areas  of 
vulnerability.  The  importance  of  T&E  in 
the  acquisition  process  is  summarized  well 
in  a  December  1986  report  produced  by  the 
General  Accounting  Office.  While  the  fol¬ 
lowing  remarks  focus  on  OT&E,  they  also 
serve  to  underscore  the  importance  of  the 
T&E  process  as  a  whole; 
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OT&E  is  the  primary  means  of  assess¬ 
ing  weapon  system  performance. 
OT&E  results  are  important  in  making 
key  decisions  in  the  acquisition  pro¬ 
cess,  especially  the  decision  to  proceed 
from  development  to  production. 
OT&E  results  provide  an  indication  of 
how  well  new  systems  will  work  and 
can  be  invaluable  in  identifying  inef¬ 
fective  or  unreliable  systems  before 
they  are  produced. 

Starting  production  before  adequate 
OT&E  is  completed  has  some  risks.  If 
adequate  OT&E  is  not  done  and  the 


weapon  system  does  not  perform  sat¬ 
isfactorily  in  the  field,  significant 
changes  may  be  required.  Moreover, 
the  changes  will  not  be  limited  to  a  few 
developmental  models,  but  may  also 
be  applied  to  items  already  produced 
and  deployed.  In  extreme  situations, 
DoD  also  risks  (1)  deploying  systems 
which  cannot  adequately  perform  sig¬ 
nificant  portions  of  their  missions,  thus 
degrading  our  deterrent/defensive 
capabilities  and  (2)  endangering  the 
safety  of  military  personnel  who  oper¬ 
ate  and  maintain  the  systems. 
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2 

THE  TEST  AND  EVALUATION 
PROCESS 


2.1  INTRODUCTION 

The  fundamental  purpose  of  test  and  evalu¬ 
ation  (T&E)  in  a  defense  system's  develop¬ 
ment  and  acquisition  program  is  to  identify 
the  areas  of  risk  to  be  reduced  or  elimi¬ 
nated.  During  the  early  phases  of  develop¬ 
ment,  T&E  is  conducted  to  demonstrate  the 
feasibility  of  conceptual  approaches,  evalu¬ 
ate  design  risk,  identify  design  alternatives, 
compare  and  analyze  trade-offs,  and  esti¬ 
mate  satisfaction  of  operational  require¬ 
ments.  As  a  system  undergoes  design  and 
development,  the  emphasis  in  testing 
moves  gradually  from  development  test 
and  evaluation  (DT&E),  which  is  concerned 
chiefly  with  attainment  of  engineering  de¬ 
sign  goals,  to  operational  test  and  evalua¬ 
tion  (OT&E),  which  focuses  on  questions  of 
operational  effectiveness,  suitability  and 
supportability.  Although  there  are  usually 
separate  development  and  operational  test 
events,  DT&E  and  OT&E  are  not  necessar¬ 
ily  serial  phases  in  the  evolution  of  a  weapon 
system.  Combined  and  concurrent  devel¬ 
opment  and  operational  testing  is  encour¬ 
aged  when  appropriate  (Reference  16). 

Test  and  evaluation  has  its  origins  in  the 
testing  of  hardware;  this  tradition  is  heavily 
embedded  in  its  vocabulary  and  proce¬ 
dures.  The  advent  of  software-intensive 
systems  has  brought  new  challenges  and 
new  approaches  to  testing,  which  are  dis¬ 
cussed  in  Chapter  18  of  this  management 
guide.  Remaining  constant  throughout  the 
T&E  process,  whether  testing  hardware  or 


software,  is  the  need  for  thorough,  logical, 
systematic  and  early  test  planning  includ¬ 
ing  feedback  of  well-documented  and  un¬ 
biased  T&E  results  to  system  developers, 
users  and  decision-makers. 

Test  and  evaluation  has  many  useful  func¬ 
tions  and  provides  information  to  many 
customers.  The  T&E  gives  information  to: 
developers  for  identifying  and  resolving 
technical  difficulties;  decision-makers  re¬ 
sponsible  for  procuring  a  new  system  and 
for  the  best  use  of  limited  resources;  and  to 
operational  users  for  refining  requirements 
and  supporting  development  of  effective 
tactics,  doctrine  and  procedures. 

2.2  DEFENSE  SYSTEM 
ACQUISITION  PROCESS 

The  defense  system  acquisition  process  was 
revised  in  1991  to  make  it  less  costly,  less 
time-consuming  and  more  responsive  to 
the  needs  of  the  operational  community. 
As  it  is  now  structured,  the  defense  system 
life  cycle  consists  of  the  following  four 
phases: 

(1)  Concept  Exploration  and  Definition 

(2)  Demonstration  and  Validation 

(3)  Engineering  and  Manufacturing  De¬ 
velopment 

(4)  Production  and  Deployment/Opera¬ 
tional  and  Support. 
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As  Figure  2-1  shows,  these  phases  are  sepa¬ 
rated  by  key  decision  points  when  a  mile¬ 
stone  (MS)  decision  authority  reviews  a 
program  and  authorizes  advancement  to 
the  next  stage  in  the  cycle.  Thus  T&E  plan¬ 
ning  and  test  results  play  an  important  part 
in  the  milestone  review  process. 

The  following  brief  description  of  the  de¬ 
fense  system  acquisition  process  shows  how 
T&E  fits  within  the  context  of  the  larger 
process.  The  description  is  based  primarily 
upon  information  found  in  DOD  Instruc¬ 
tion  5000.2  (Reference  16). 

2.2.1  Concept  Exploration 
and  Definition  (Phase  0) 

The  defense  system  acquisition  process 
begins  with  the  submission  of  a  Mission 
Need  Statement.  A  Concept  Exploration 
and  Definition  Phase  follows  the  Milestone 
0  decision  during  which  alternative  ap¬ 
proaches  for  satisfying  the  requirement  are 
investigated.  The  Concept  Exploration  and 
Definition  Phase  concludes  with  the  Mile¬ 
stone  I  selection  of  a  concept  or  concepts  to 
enter  a  Demonstration  and  Validation 
Phase.  The  Milestone  I  decision  establishes 
broad  objectives  for  program  cost,  sched¬ 
ule,  operational  effectiveness  and  suitabil¬ 
ity.  Key  documents  for  the  T&E  manager  at 
the  time  of  the  Milestone  I  review  are  the 
Acquisition  Decision  Memorandum  (ADM) 
(exit  criteria).  Integrated  Program  Summary 
(IPS),  Operational  Requirement  Document 
(ORD),  Mission  Need  Statement  (MNS), 
System  Threat  Assessment  Report  (STAR), 
the  Test  and  Evaluation  Master  Plan 
(TEMP),  and  the  Integrated  Logistics  Sup¬ 
port  Plan  (ILSP) /Logistics  Support  Analy¬ 
sis  (LSA).  Additional  program  manage¬ 
ment  documents  prepared  before  Milestone 
I  include:  the  Cost  and  Operational  Effec¬ 
tiveness  Analysis  (COEA),  Independent 
Cost  Estimate,  and  Concept  Baseline,  which 
summarizes  the  weapon's  functional 


specifications,  performance  parameters, 
and  cost  and  schedule  objectives. 

2.2.2  Demonstration 
and  Validation  (Phase  I) 

After  the  Milestone  I  decision  for  a  pro¬ 
gram  start,  the  Demonstration  and  Valida¬ 
tion  Phase  begins  during  which  selected 
concepts,  typically  brassboard  or  early  pro¬ 
totype,  are  refined  through  engineering  and 
analysis.  This  phase  ends  with  the  Mile¬ 
stone  II  decision  to  either  enter  into  engi¬ 
neering  and  manufacturing  development 
(EMD)  or  terminate  the  program.  The  Mile¬ 
stone  II  decision  establishes  more  specific 
cost,  schedule,  operational  effectiveness  and 
suitability  objectives  and  thresholds.  Docu¬ 
ments  interesting  to  the  T&E  manager  at 
the  time  of  the  Milestone  II  review  include 
the  ADM  (exit  criteria),  IPS,  updated  TEMP, 
COEA,  updated  ORD,  Development 
Baseline,  the  early  Operational  Assessment 
and  low-rate  initial  production  (LRIP)  guid¬ 
ance. 

2.2.3  Engineering  and  Manufacturing 
Development  (Phase  II) 

During  the  EMD  Phase,  the  selected  sys¬ 
tem  and  its  principal  items  of  support  are 
fabricated  as  engineer  development  mod¬ 
els.  This  phase  ends  with  the  Milestone  III 
decision  to  enter  full-rate  production  of  the 
system.  Key  documents  for  the  T&E  man¬ 
ager  at  the  time  of  the  Milestone  III  review 
are  the  IPS,  updated  TEMP,  Beyond  LRIP 
Report,  and  Live  Fire  Test  Report.  The  di¬ 
rector  of  OT&E  is  required  by  law  to  docu¬ 
ment  his  assessment  of  the  adequacy  of 
OT&E  and  the  reported  operational  effec¬ 
tiveness  and  suitability  of  the  system.  This 
is  done  in  the  Beyond  LRIP  Report.  Also 
mandated  by  law  is  the  requirement  for  the 
director  of  T&E  to  write  the  Live  Fire  Test 
Report  prior  to  proceeding  beyond  LRIP. 
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Figure  2-1 .  Testing  and  the  Acquisition  Process 


2.2.4  Production/Deployment 
and  Operations/Support  (Phase  III) 

The  Milestone  III  decision  is  followed  by  a 
Full-Rate  Production /Deployment  Phase. 
This  phase  ends  with  a  Major  Modification 
Approval  (Milestone  IV)  to  identify  the 
actions  and  resources  needed  to  achieve 
and  maintain  operational  readiness  and 
support  objectives.  To  determine  whether 
major  upgrades  are  necessary  or  deficien¬ 
cies  warrant  consideration  of  replacement, 
the  Milestone  FV  decision  encompasses  a 
review  of  system  operational  effectiveness, 
suitability  and  readiness.  In  preparation 
for  Milestones  IV,  the  IPS,  TEMP,  and  pro¬ 
duction  baseline  are  updated  to  describe 
the  program  status,  changes  and  issues. 

2.3  T&E  AND  THE  SYSTEMS 
ENGINEERING  PROCESS 

In  the  early  1970s,  Department  of  Defense 
(DOD)  test  policy  became  more  formalized 
and  placed  greater  emphasis  on  T&E  as  a 
continuing  function  throughout  the  acqui¬ 
sition  cycle.  These  policies  stressed  the  use 
of  T&E  to  reduce  acquisition  risk  and  pro¬ 
vide  early  and  continuing  estimates  of  sys¬ 
tem  operational  effectiveness  and  opera¬ 
tional  suitability.  To  meet  these  objectives, 
appropriate  test  activities  had  to  be  fully 
integrated  into  the  overall  development 
process.  From  a  systems  engineering  per¬ 
spective,  test  planning,  testing  and  analy¬ 
sis  of  test  results  are  integral  parts  of  the 
basic  product  definition  process. 

In  MIL-STD-499,  systems  engineering  is 
defined  in  the  DOD  context:  "Systems  en¬ 
gineering  is  an  interdisciplinary  approach 
to  evolve  and  verify  an  integrated  and  op¬ 
timally  balanced  set  of  product  and  process 
designs  that  satisfy  user  needs  and  provide 
information  for  management  decision  mak¬ 
ing."  (Figure  2-2) 


A  system's  life  cycle  begins  with  the  user's 
needs,  which  are  expressed  as  constraints, 
and  the  capability  requirements  needed  to 
satisfy  mission  objectives.  Systems  engi¬ 
neering  is  essential  in  the  earliest  planning 
period,  in  conceiving  the  system  concept 
and  defining  performance  requirements  for 
system  elements.  As  the  detailed  design  is 
prepared,  systems  engineers  ensure  bal¬ 
anced  influence  of  all  required  design  spe¬ 
cialties,  including  "testability."  They  re¬ 
solve  interface  problems,  perform  design 
reviews,  perform  trade-off  analyses  and 
assist  in  verifying  performance. 

The  days  when  one  or  two  individuals 
could  design  a  complex  system,  especially 
a  huge,  modem-age  weapon  system,  are 
past.  Now  systems  are  too  complex  for  a 
small  number  of  generalists  to  accommo¬ 
date;  they  require  too  much  in-depth  knowl¬ 
edge  over  a  broad  range  of  areas  and  tech¬ 
nical  disciplines.  System  engineers  coordi¬ 
nate  the  many  specialized  engineers  in¬ 
volved  in  the  concurrent  engineering  pro¬ 
cess  and  are  responsible  for  the  integration 
of  the  components  into  a  system. 

Through  interdisciplinary  integration  sys¬ 
tems  engineering  manages  the  progress  of 
product  definition  from  system  level,  to 
configuration-item  level,  detailed  level, 
deficiency  correction,  and  modifications/ 
product  improvements.  Test  results  pro¬ 
vide  feedback  to  analyze  the  design 
progress  toward  performance  goals.  Tools 
of  systems  engineering  include  design  re¬ 
views,  configuration  management,  simu¬ 
lation,  technical  performance  measurement, 
trade-off  analysis  and  specifications. 

What  products  are  produced  by  systems 
engineering?  It  determines  what  special¬ 
ists  are  required,  what  segments  and 
nondevelopmental  items  are  used,  design 
performance  limits,  trade-off  criteria,  how 
to  test,  when  to  test,  how  to  document 
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Figure  2-2.  The  Systems  Engineering  Process 


(specifications),  and  what  management 
controls  to  apply  (technical  performance 
measurement  and  design  reviews). 

Development  testing  (DT)  and  operational 
testing  (OT)  support  the  technical  reviews 
used  to  monitor  the  systems  engineering 
process.  More  information  on  the  reviews 
is  contained  in  Chapter  8. 

2.3.1  The  Systems  Engineering  Process 

The  systems  engineering  process  is  the  it¬ 
erative  logical  sequence  of  analysis,  de¬ 
sign,  test  and  decision  activities  that  trans¬ 
forms  an  operational  need  into  the  descrip¬ 
tions  required  for  production  and  fielding 
of  all  operational  and  support  system  ele¬ 
ments.  This  process  consists  of  four  activi¬ 
ties.  They  include  functional  analysis,  syn¬ 
thesis,  evaluation  and  decision  (trade-off) 
and  description  of  system  elements. 

The  functional  analysis  activity  identifies 
what  the  system,  component  or  part  must 
do.  It  works  from  the  top  downward  ensur¬ 
ing  requirements  traceability  and  reveal¬ 
ing  alternative  concepts.  This  is  done  with¬ 
out  assuming  how  functions  will  be  accom¬ 
plished.  The  product  is  a  series  of  alterna¬ 
tive  Functional  Flow  Block  Diagrams 
(FFBD).  A  functional  analysis  can  be  ap¬ 
plied  at  every  level  of  development.  At  the 
system  level,  it  may  be  a  contractor  or  Ser¬ 
vice  effort.  During  Concept  Explora¬ 
tion  Phase,  developrneii  ial  testers  assist  the 
functional  analysis  activity  to  help  deter¬ 
mine  what  each  component's  role  will  be  as 
part  of  the  system  being  developed. 

The  synthesis  activity  involves  invention 
— conceiving  ways  to  do  each  FFBD  task — 
to  answer  the  "how"  question.  Next,  the 
physical  interfaces  implied  by  the  "how" 
answers,  are  carefully  identified  (topologi¬ 
cal  or  temporal).  The  answers  must  reflect 
all  technology  selection  factors.  Synthesis 
tools  include  Requirements  Allocation 


Sheets  (RAS),  which  translate  functional 
statements  into  design  requirements  and 
permit  a  long  and  complex  interactive  in¬ 
vention  process  with  control,  visibility  and 
requirements  traceability.  Developmental 
testers  conduct  demonstration/validation 
testing  to  determine  how  the  components 
will  perform  assigned  functions  to  assist 
this  synthesis  activity. 

The  evaluation  and  decision  activity  is  a 
trade-off  of  alternative  approaches  to 
"how."  This  activity  is  conducted  in  accor¬ 
dance  with  decision  criteria  set  by  higher- 
level  technical  requirements  for  such  things 
as  life-cycle  costs,  effectiveness,  reliability, 
availability,  maintainability,  risk  limits, 
schedule,  etc.  It  is  repeated  at  each  level  of 
development.  The  evaluation  and  decision 
activity  is  assisted  by  developmental  testers 
during  the  later  Demonstration  and  Vali¬ 
dation  Phase  and  the  Engineering  and 
Manufacturing  Development  Phase  when 
competitive  testing  between  alternative 
approaches  is  performed. 

The  final  activity  is  a  description  of  system 
elements.  Developing  as  the  result  of  previ¬ 
ous  activities  and  as  the  final  system  design 
is  determined,  this  activity  takes  form  when 
specifications  are  verified  through  testing 
and  when  reviewed  in  the  Physical  Con¬ 
figuration  and  Functional  Configuration 
Audits.  During  the  EMD  Phase,  operational 
testers  assist  in  this  activity.  They  conduct 
operational  testing  of  the  test  items/sys¬ 
tems  to  help  determine  the  personnel, 
equipment,  facilities,  software  and  techni¬ 
cal  data  requirements  of  the  new  system 
when  used  by  typical  military  personnel. 
Figure  2-2,  System  Engineering  Process, 
depicts  the  activities  and  their  interactions. 

2.3.2  The  System  Engineering 
Management  Plan 

The  System  Engineering  Management  Plan 
(SEMP)  is  a  concise,  top-level  management 
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lESTING  DEFINED  IN  PERFORMED  AGAINST  PARTICIPANTS 


Figure  2-3.  Systems  Engineering  and  Test  and  Evaluation 


plan  for  the  integration  of  all  system  design 
activities.  Its  purpose  is  to  make  visible  the 
organization;  mechanisms  for  direction  and 
control;  and  personnel  for  the  attainment 
of  cost,  performance  and  schedule  objec¬ 
tives.  The  SEMP  defines  and  describes  the 
type  and  degree  of  system  engineering 
management,  the  system  engineering  pro¬ 
cess,  and  the  integration  of  related  engi¬ 
neering  programs.  The  design  evolution 
process,  which  is  described  in  the  SEMP, 
forms  the  basis  for  comprehensive  test  and 
evaluation  planning. 

The  TEMP  must  be  consistent  with  the 
SEMP.  The  testing  program  outlined  in  the 
TEMP  must  provide  the  technical  perfor¬ 
mance  measurements  data  required  for  all 
design  decision  points,  audits  and  reviews 
that  are  a  part  of  the  system's  engineering 
process  outlined  in  the  SEMP.  The  configu¬ 
ration  management  process  outlined  in  the 
SEMP  controls  the  baseline  for  the  test  pro¬ 
grams  and  incorporates  design  modifica¬ 
tions  to  the  baseline  determined  to  be  nec¬ 
essary  by  T&E. 

The  TEMP  and  the  SEMP  must  be  traceable 
to  each  other.  The  system  description  in  the 
TEMP  must  be  traceable  to  systems  engi¬ 
neering  documentation  such  as  the  FFBDs, 
the  RASs,  and  the  Test  Requirements  Sheets 
(TRSs).  Key  functions  and  interfaces  of  the 
system  with  other  systems  must  be  de¬ 
scribed  and  correlated  with  the  systems 
engineering  documentation  and  the  sys¬ 
tem  specification  (Type  A).  Operational  and 
technical  thresholds  in  the  SEMP  include 
specific  performance  requirements  that 
become  test  planning  limits.  They  must  be 
traceable  through  the  planned  systems  en¬ 
gineering  documentation  and  can  be  corre¬ 
lated  to  the  content  of  the  Technical  Perfor¬ 
mance  Measurement  (TPM)  Program.  Fail¬ 
ure  criteria  for  reliability  thresholds  during 
OT&E  testing  must  be  delineated  and 
agreed  upon  by  the  program  manager  and 


the  operational  test  director  and  reflected 
in  the  SEMP  and  the  TEMP. 

2.3.3  Technical  Performance 
Measurement 

Technical  performance  measurement  iden¬ 
tifies  critical  te<'’">ical  parameters  that  are 
at  risk  during  design.  It  tracks  evaluation 
and  test  data,  makes  predictions  about 
whether  the  parameter  can  achieve  final 
technical  success  within  the  allocated  re¬ 
sources,  and  assists  in  managing  the  tech¬ 
nical  program. 

The  TPM  Program  is  an  integral  part  of  the 
T&E  program.  The  TPM  is  defined  as  prod¬ 
uct  design  assessment  and  forms  the  back¬ 
bone  of  the  development  testing  program. 
It  estimates,  through  engineering  analyses 
and  tests,  the  values  of  essential  perfor¬ 
mance  parameters  of  the  current  program 
design.  It  serves  as  a  major  input  in  the 
continuous  overall  evaluation  of  opera¬ 
tional  effectiveness  and  suitability.  Design 
reviews  are  conducted  to  measure  systems 
engineering  progress.  For  more  informa¬ 
tion,  see  Chapter  8.  Figure  2-4  depicts  the 
technical  reviews  that  usually  take  place 
during  the  systems  engineering  process 
and  the  related  specification  documents. 

2.3.4  Product  Baselining  and  T&E 

The  systems  engineering  process  estab¬ 
lishes  a  product  baseline  throughout  the 
acquisition  cycle.  This  baseline  can  be  modi¬ 
fied  with  the  results  of  engineering  and 
testing.  The  testing  used  to  prove  the  tech¬ 
nical  or  development  baseline  is  rarely  the 
same  as  the  operational  testing  or  produc¬ 
tion  baseline. 

Related  to  the  product  baseline  is  the  pro¬ 
cess  of  configuration  management.  Con¬ 
figuration  management  benefits  the  test 
and  evaluation  community  in  two  ways. 
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Figure  2-4.  Design  Reviews 


Through  configuration  management,  the 
baseline  product  to  be  used  for  testing  is 
determined.  Also,  changes  that  occur  to  the 
baseline  as  a  result  of  testing  and  design 
reviews  are  incorporated  into  the  test  ar¬ 
ticle  before  the  new  phase  of  testing  (to 
prevent  retest  of  a  bad  design). 

2.4  DEFINITIONS 

Test  and  evaluation  is  the  deliberate  and 
rational  generation  of  data,  which  concerns 
the  nature  of  the  emerging  system,  and  the 
creation  of  information  useful  to  the  techni¬ 
cal  and  managerial  personnel  controlling 
its  development.  In  the  broad  sense,  T&E 
may  be  defined  as  all  physical  testing,  mod¬ 
eling,  simulation,  experimentation  and  re¬ 
lated  analyses  performed  during  research, 
development,  introduction  and  employ¬ 
ment  of  a  weapon  system  or  subsystem. 
The  Glossary:  Defense  Acquisition  Acronyms 
and  Terms,  produced  by  the  Defense  Sys¬ 
tems  Management  College,  September 
1991,  defines  "Test"  and  "Test  and  Evalua 
tion"  as  follows: 

A  "test"  is  any  program  or  procedure 
which  is  designed  to  obtain,  verify,  or 
provide  data  for  the  evaluation  of:  re¬ 
search  and  development  (other  than 
laboratory  experiments);  progress  in 
accomplishing  development  objec¬ 
tives;  or  performance  and  operational 
capability  of  systems,  subsystems, 
components,  and  equipment  items. 

"Test  and  Evaluation"  is  the  process 
by  which  a  system  or  components  are 
compared  against  requirements  and 
specifications  through  testing.  The  re¬ 
sults  are  evaluated  to  assess  progress 
of  design,  performance,  supportabil- 
ity,  etc.  There  are  three  types  of  T&E- 
Development  (DT&E),  Operational 
(OT&E),  and  Production  Acceptance 


(PAT&E)  —  occurring  during  the  ac¬ 
quisition  cycle.  DT&E  is  conducted  to 
assist  the  engineering  design  and  de¬ 
velopment  process  and  to  verify  at¬ 
tainment  of  technical  performance 
specifications  and  objectives.  OT&E  is 
conducted  to  estimate  a  system's  op¬ 
erational  effectiveness  and  suitability, 
identify  needed  modifications,  and 
provide  information  on  tactics,  doc¬ 
trine,  organization,  and  personnel  re¬ 
quirements.  PAT&E  is  conducted  on 
production  items  to  demonstrate  that 
those  items  meet  the  requirements  and 
specifications  of  the  procuring  con¬ 
tracts  or  agreements.  OT&E  is  further 
subdivided  into  two  phases-  Initial 
Operational  (lOT&E)  and  Follow-on 
Operational  (FOT&E).  lOT&E  must  be 
conducted  before  the  production  deci¬ 
sion  (MS  III)  to  provide  a  credible  esti¬ 
mate  of  operational  effectiveness  and 
suitability.  Therefore,  lOT&E  must  be 
conducted  on  a  system  as  close  to  a 
production  configuration  as  possible, 
in  an  operationally  realistic  environ¬ 
ment,  by  typical  user  personnel. 
FOT&E  is  conducted  on  the  deployed 
system  to  determine  if  operational  ef¬ 
fectiveness  and  suitability  is,  in  fact, 
being  attained. 

2.5  SUMMARY 

Test  and  evaluation  is  an  engineering  tool 
used  to  reduce  risk  throughout  the  defense 
system  acquisition  cycle.  This  cycle  con¬ 
sists  of  five  phases  separated  by  discrete 
milestones.  The  T&E  results  are  used  to 
support  the  design  reviews  that  form  an 
important  part  of  the  system  engineering 
process  used  by  system  developers  and  to 
aid  in  the  milestone  decision  process  used 
by  senior  decision  authorities  in  the  De¬ 
partment  of  Defense. 
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3 

T&E  POLIO  STRUCTURE 
AND  OVERSIGHT  MECHANISM 


3.1  INTRODUCTION 

This  chapter  provides  an  overview  of  the 
policy  and  organizations  that  govern  the 
conduct  of  test  and  evaluation  (T&E)  ac¬ 
tivities  within  the  Department  of  Defense 
(DOD)  and  discusses  congressional  legis¬ 
lation  and  activities  for  compliance  by  the 
DOD.  It  outlines  the  responsibilities  of  DOD 
test  organizations,  at  the  Office  of  the  Sec¬ 
retary  of  Defense  (OSD)  and  Service  levels, 
and  describes  related  T&E  policy. 

3.2  THE  CONGRESS 

The  Congress  has  shown  a  long-standing 
interest  in  influencing  the  DOD  acquisition 
process.  During  the  early  1970s,  in  response 
to  urging  by  the  Congress  and  recommen¬ 
dations  by  a  Presidential  Blue  Ribbon  Panel 
on  Defense  Management,  the  Deputy  Sec¬ 
retary  of  Defense,  David  Packard,  promul¬ 
gated  a  package  of  policy  initiatives  that 
established  the  Defense  Systems  Acquisi¬ 
tion  Review  Coimcil  (DSARC).  The  DSARC 
was  organized  to  resolve  acquisition  is¬ 
sues,  whenever  possible,  and  to  provide 
recommendations  to  the  Secretary  of  De¬ 
fense  (SECDEF)  on  the  acquisition  of  major 
weapon  systems.  Also,  as  a  result  of  the 
Congressional  Directives,  the  Army  and 
Air  Force  established  independent  opera¬ 
tional  test  agencies.  The  Navy  Operational 
Test  and  Evaluation  Force  was  established 
in  the  late  1960s.  In  1983,  similar  concerns 
led  the  Congress  to  direct  the  establish¬ 
ment  of  the  independent  Office  of  Director, 


Operational  Test  and  Evaluation  (DOT&E), 
within  the  OSD.  In  1985  a  report  released 
by  the  President's  Blue  Ribbon  Conunis- 
sion  on  Defense  Management,  chaired  by 
David  Packard,  made  significant  recom¬ 
mendations  on  the  management  and  over¬ 
sight  of  DOD's  acquisition  process  and, 
specifically,  test  and  evaluation.  All  the 
Commission's  recommendations  have  not 
been  implemented,  and  the  full  impact  of 
these  recommendations  is  not  yet  realized. 
In  FY  1987  the  Defense  Authorization  Act 
required  live  fire  testing  of  weapon  sys¬ 
tems  before  the  Production  Phase  begins. 

The  DOD  is  required  to  provide  to  the 
Congress  the  following  reports  on  test  and 
evaluation: 

•  Congressional  Data  Sheets  (CDS).  The 
CDS  are  annual  reports  on  each  major  sys¬ 
tem  acquisition.  They  must  be  updated 
before  the  contract  is  awarded  and  when 
procurement  of  the  system  is  requested  in 
the  fiscal  year.  The  CDS  describe  the  devel¬ 
opment  test  and  evaluation  (DT&E)  and 
operational  test  and  evaluation  (OT&E)  to 
be  performed  and  system  characteristics. 

•  Selected  Acquisition  Report  (SAR).  The 
SAR  describes  the  system  characteristics 
required  and  outlines  significant  progress 
and  problems  encountered.  It  lists  tests 
completed  and  issues  identified  during  test¬ 
ing. 
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•  Annual  System  Operational  Test  Re¬ 
port.  The  Annual  Systems  Operational  Test 
Report  is  provided  by  the  DOT&E  to  the 
SECDEF  and  the  committees  on  Armed 
Services  and  Appropriations.  The  report 
provides  a  narrative  and  resource  sum¬ 
mary  of  OT&E  and  OT&E-related  issues, 
activities,  and  assessments. 

•  Beyond  Low-Rate  Initial  Production 
(LRIP)  Report.  Before  proceeding  beyond 
LRIP  for  each  major  system  acquisition 
program,  the  Director,  Operational  Test 
and  Evaluation,  must  report  to  the  SECDEF 
and  the  Congress.  This  report  addresses 
the  adequacy  of  OT&E  and  whether  the 
T&E  results  confirm  that  the  tested  item  or 
component  is  effective  and  suitable  for  com¬ 
bat. 

3.3  OSD  OVERSIGHT  STRUCTURE 

The  DOD  organization  for  the  oversight  of 
T&E  is  illustrated  in  Figure  3-1.  In  the  OSD, 
T&E  oversight  is  performed  by  two  pri¬ 
mary  offices:  the  Director,  Test  and  Evalu¬ 
ation  (DTE)  and  the  Director  Operational 
Test  and  Evaluation  (DOT&E).  The  man¬ 
agement  of  acquisition  programs  in  OSD  is 
performed  by  the  Defense  Acquisition  Ex¬ 
ecutive  (DAE),  who  uses  the  Defense  Ac¬ 
quisition  Board  (DAB)  and  subcommittees 
to  process  information  for  decisions.  The 
Under  Secretary  of  Defense  for  Acquisition 
(USD(A))  uses  the  DAB  and  its  committees 
to  provide  the  senior-level  decision  process 
for  the  acquisition  of  weapon  systems. 

3.3.1  Defense  Acquisition 
Executive  (DAE) 

The  DAE  position,  established  in  Septem¬ 
ber  1986,  is  held  by  the  USD(A).  The  re¬ 
sponsibilities  include  "establishing  policies 
for  acquisition  (including  procurement,  re¬ 
search  and  development,  logistics,  devel¬ 
opment  testing,  and  contracts  administra¬ 
tion)  for  all  elements  of  the  Department  of 


Defense."  His  charter  includes  the  author¬ 
ity  over  the  Service  and  defense  agencies 
on  policy,  procedure  and  execution  of  the 
acquisition  process. 

3.3.2  Defense  Acquisition  Board  (DAB) 

The  DAB  is  the  primary  forum  used  by 
OSD  to  provide  advice,  assistance  and  rec¬ 
ommendations,  and  to  resolve  issues  re¬ 
garding  all  operating  and  policy  aspects  of 
the  DOD  acquisition  system.  The  DAB  is 
the  senior  management  acquisition  board 
chaired  by  the  DAE  and  attended  by  the 
Vice  Chairman  of  Joint  Chiefs  of  Staff,  Prin¬ 
cipal  Under  Secretary  of  Defense  for  Ac¬ 
quisition  and  the  component  acquisition 
executives.  The  DAB  conducts  business 
through  working  committees  (DODI 
5000.2). 

3.3.3  Defense  Planning 
Resources  Board  (DPRB) 

The  DPRB  was  established  by  the  SECDEF 
in  1979  to  advise  the  SECDEF  on  policy, 
planning,  program  and  budget  issues.  The 
DPRB  is  chaired  by  the  Deputy  Secretary  of 
Defense  and  is  responsible  for  the  manage¬ 
ment  and  oversight  of  all  aspects  of  the 
DOD  planning,  programming  and  budget¬ 
ing  process.  It  oversees  the  budget  reviews 
process  and,  therefore,  has  a  major  impact 
on  test  and  evaluation  resources. 

3.3.4  Director  Test  and  Evaluation  (DTE) 

The  DTE  serves  as  the  principal  staff  assis¬ 
tant  and  advisor  to  the  USD(A)  for  T&E 
matters.  He  has  authority  and  responsibil¬ 
ity  for  all  DT&E  conducted  on  designated 
major  programs.  The  DTE  organization  is 
illustrated  in  Figure  3-1. 

3.3.4.1  Duties  of  the  DTE 

Within  the  acquisition  community,  the  DTE: 

•  Serves  as  the  focal  point  for  coordina- 


3-2 


SECRETARY  Of  DEFENSE 


Figure  3-1 .  DOD  Test  and  Evaluation  Organization 


tion  of  all  major  program  test  and  evalua¬ 
tion  master  plans  (TEMPs).  Signs  for  ap¬ 
proval  of  DT&E  portion  of  TEMPs; 

•  Reviews  major  defense  acquisition  pro¬ 
gram  documentation  for  DT&E  implica¬ 
tions  and  resource  requirements  to  provide 
comments  to  the  USD(A),  DAE  or  DAB; 

•  Observes  DT&E  to  ensure  adequacy  of 
testing  and  to  assess  test  results; 

•  Provides  the  DAE  and  DAB  with  a 
technical  assessment  of  development  T&E 
conducted  on  a  weapon  system; 

•  Provides  advice  and  makes  recommen¬ 
dations  to  the  SECDEF  and  issues  guidance 
to  the  component  acquisition  executives 
with  respect  to  DT&E; 

•  Performs  the  administrative  process¬ 
ing  of  nominations  and  charters  for  joint 
development  test  programs; 

•  Provides  management  oversight  of  the 
Major  Range  and  Test  Facility  Base 
(MRTFB); 

•  Administers  the  Foreign  Weapons 
Evaluation  Program  and  NATO  Compara¬ 
tive  Test  Program; 

•  Confirms,  with  advice  from  the  Assis¬ 
tant  to  the  Secretary  of  Defense  (Atomic 
Energy),  that  nuclear  survivability  and 
hardness  objectives  have  been  addressed 
during  DT&E; 

•  Responsible  for  the  Live  Fire  Test  Pro¬ 
gram. 

3.3.4.2  DTE  and  Service  Reports 

During  the  testing  of  major  and  designated 
weapon  systems,  the  DTE  and  Services  in¬ 
teraction  includes  the  following  reporting 
requirements: 


•  A  TEMP  (either  initial  or  updated,  as 
appropriate)  must  be  provided  for  consid¬ 
eration  and  approval  before  each  milestone 
review,  starting  with  Milestone  (MS)  I. 

•  A  significant  T&E  Event  Report  must 
be  provided  to  the  DTE  within  24  hours  of 
the  test  event. 

•  An  End-of-Test  Phase  Report  must  be 
provided  to  the  DTE  listing  the  T&E  re¬ 
sults,  conclusions  and  recommendations 
prior  to  a  milestone  decision  or  the  final 
decision  to  proceed  beyond  LRIP. 

3.3.5  Director  Operational  Test 
and  Evaluation  (DOT&E) 

As  illustrated  in  Figure  3-1,  the  director 
reports  directly  to  the  SECDEF  and  has 
special  reporting  requirements  to  the  Con¬ 
gress.  The  DOT&E's  responsibility  to  the 
Congress  is  to  provide  an  unbiased  win¬ 
dow  of  insight  into  the  operational  effec¬ 
tiveness  and  suitability  of  new  weapon 
systems. 

3.3.5.1  Duties  and  Functions 
of  the  DOT&E 

The  specific  duties  of  DOT&E  are  outlined 
in  DOD  Directive  5141.2.  The  functions  of 
the  office  include: 

•  Obtaining  reports,  information,  advice 
and  assistance  as  necessary  to  carry  out 
assigned  functions  (DOT&E  has  access  to 
all  records  and  data  in  DOD  on  acquisition 
programs); 

•  Signing  the  TEMPs  for  approval  of 
OT&E  and  approving  the  OT&E  funding 
for  major  systems  acquisition; 

•  Approving  test  plans  on  all  major  sys¬ 
tems  prior  to  system  starting  operational 
testing  (approval  in  writing  required  be¬ 
fore  operational  testing  may  begin); 
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•  Providing  observers  during  prepara¬ 
tion  and  conduct  of  OT&E; 

•  Analyzing  results  of  OT&E  conducted 
for  each  major  or  designated  defense  acqui¬ 
sition  program  and  submitting  a  report  to 
the  SECDEF  and  the  Congress  on  the  ad¬ 
equacy  of  the  operational  test  and  evalua¬ 
tion  performed; 

•  A  final  decision  to  proceed  \vith  a  major 
program  beyond  TRIP  cannot  be  made  until 
EXDT&E  has  reported  (Beyond  TRIP  Re¬ 
port)  to  the  SECDEF  and  to  congressional 
Committees  on  Armed  Services  and  Ap¬ 
propriations  on  the  adequacy  of  T&E  and 
whether  the  results  confirm  the  system's 
operational  effectiveness  and  suitability. 

3.3.5.2  DOT&E  and  Service  Interactions 

For  DOD  and  DOT&E-designated  acquisi¬ 
tion  programs,  the  Service  provides  the 
DOT&E  the  following: 

•  A  draft  copy  of  the  Operational  Test 
Plan  for  review; 

•  Significant  Test  Plan  changes; 

•  The  final  Service  lOT&E  report  must  be 
submitted  to  DOT&E  before  the  DAB  Mile¬ 
stone  III  review. 

3.4  SERVICE  T&E 
MANAGEMENT  STRUCTURES 

3.4.1  Army  T&E  Organizational 
Relationship 

The  Army  management  structure  for  T&E 
is  illustrated  in  Figure  3-2. 

3.4.1.1  Army  Acquisition  Executive 

The  Under  Secretary  of  the  Army  is  the 
Army  Acquisition  Executive  (AAE).  The 
AAE  is  responsible  for  all  acquisition  T&E 


(operational  and  developmental  tests)  plan¬ 
ning,  programming,  budgeting,  and  devel¬ 
opmental  testing  policy  and  oversight.  The 
AAE  performs  these  duties  with  the  assis¬ 
tance  of  the  Assistant  Secretary  of  the  Army, 
Research,  Development,  and  Acquisition 
(ASA/RDA).  As  illustrated  in  Figure  3-2, 
the  ASA  /  RDA  is  o  rganized  to  provide  tech¬ 
nical  assessments  and  program  evaluations. 
He  resolves  acquisition  issues  whenever 
possible  and  recommends  acquisition  of 
weapon  systems  to  the  AAE.  The  Deputy 
Under  Secretary  of  the  Army  for  Opera¬ 
tions  Research  (DUSA(OR))  is  chartered  to 
supervise  all  Army  T&E  policy  and  has 
oversight  for  all  Army  T&E. 

3.4.1.2  Army  Technical  Testers 
and  Evaluators 

The  U.S.  Army  Materiel  Command  (AMC) 
is  responsible  for  the  management  of  DT&E. 
The  Test  and  Evaluation  Command 
(TECOM)  has  the  primary  responsibility 
for  conducting  technical  tests  for  the  Army 
and  under  certain  conditions  Army  Mate¬ 
riel  Systems  Analysis  Agency  (AMSA.A) 
conducts  the  evaluation.  The  TECOM  is 
responsible  for: 

•  Planning,  executing  and  reporting  the 
results  of  technical  tests.  Technical  tests 
include  development  tests,  technical  feasi- 
bilit'^  tests,  production  qualification  tests, 
joint  tests  and  contractor/ foreign  tests; 

•  Providing  test  facilities  and  technical 
expertise  in  support  of  the  T&E  life  cycle; 

•  Maintaining  the  Army's  Major  Range 
and  Test  Facility  Base; 

•  Maintaining  the  Army's  T&E  data  base; 

•  Researching,  developing  and  acquir¬ 
ing  instrumentation  and  developing  new 
and  improved  test  methodology; 
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Figure  3-2.  Army  T&E  Organization 


•  Providing  safety  confirmations. 

3.4.1.3  Army  Operational  Test 
and  Evaluation  Command 

•  The  Army  Operational  Test  and  Evalu¬ 
ation  Command  (OPTEC)  is  responsible 
for  the  management  of  operational  testing 
as  well  as  the  management  of  joint  user 
testing.  The  OPTEC  is  an  independent 
agency  reporting  directly  to  the  Army  Vice 
Chief  of  Staff. 

•  The  OPTEC  combines  the  evaluation 
function  performed  by  the  Operational 
Evaluation  Command  (OEC)  and  the  op¬ 
erational  testing  function  performed  by  the 
Test  and  Experimentation  Command 
(TEXCOM). 

•  The  U.S  Army  Forces  Command 
(FORSCOM)  supports  testing  by  provid¬ 
ing  user  troops  and  facilities  as  needed. 

3.4.2  Navy  T&E  Organizational 
Relationship 

The  organizational  structure  for  T&-E  in  the 
Navy  is  illustrated  in  Figure  3-3.  Within  the 
Navy  Secretariat,  the  Secretary  of  the  Navy 
has  assigned  general  and  specific  research, 
development,  test  and  evaluation  (RDT &E) 
responsibilities  to  the  Assistant  Secretary 
of  the  Navy  (Research,  Development  and 
Acquisition)  and  to  the  Chief  of  Naval 
Operations  (CNO).  The  CNO  has  responsi¬ 
bility  for  ensuring  the  adequacy  of  the 
Navy's  overall  test  and  evaluation  pro¬ 
gram.  The  T&E  policy  and  guidance  are 
exercised  through  the  Directorate  of  Navy; 
T&E  and  Technology  Requirements  (N-91) 
staff  support  is  provided  by  the  Test  and 
Evaluation  Division  (N-912)  which  has  cog¬ 
nizance  over  planning,  conducting  and  re¬ 
porting  all  T&E  associated  with  develop¬ 
ment  of  systems. 


3.4.2.2  Navy  DT&E  Organizations 

The  Navy's  senior  systems  development 
authority  is  divided  among  the  command¬ 
ers  of  the  system  commands  with  N  A  V  AIR 
developing  and  performing  DT&E  on  air¬ 
craft,  NAVSEA  developing  and  perform¬ 
ing  DT&E  on  ships  and  SPA  WAR  develop¬ 
ing  and  performing  DT&E  on  all  other  sys¬ 
tems.  System  acquisition  is  controlled  by  a 
chartered  program  manager  or  by  the  com¬ 
mander  of  a  systems  command.  In  both 
cases,  the  designated  developing  agency  is 
responsible  for  DT&E  and  for  the  coordina¬ 
tion  of  all  test  and  evaluation  planning  in 
the  TEMP.  Developing  Agencies  (DAs)  are 
responsible  for: 

•  Developing  test  issues  based  on  the 
thresholds  established  by  the  ustr  in  the 
Operational  Requirements  Document; 

•  Identifying  the  testing  facilities  and 
resources  required  to  conduct  the  DT&E; 

•  Developing  the  DT&E  test  reports  and 
quick-look  reports. 

3.4.2.3  Navy  Operational  Test 
and  Evaluation  Force 

The  Commander  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR)  com¬ 
mands  the  Navy's  independent  operational 
test  and  evaluation  activity  and  reports 
directly  to  the  CNO.  The  functions  of  the 
COMOPTEVFOR  include: 

•  Establishing  early  liaison  with  the  DA 
to  ensure  an  understanding  of  the  test  re¬ 
quirements  and  plans; 

•  Reviewing  acquisition  program  docu¬ 
mentation  to  ensure  that  documents  are 
adequate  to  support  a  meaningful  T&E 
program; 
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•  Planning  and  conducting  realistic 
OT&E; 

•  Developing  tactics  and  procedures  for 
the  employment  of  systems  that  undergo 
OT&E  (as  directed  by  the  CNO); 

•  Providing  recommendations  to  the 
CNO  for  the  development  of  new  capabili¬ 
ties  or  the  upgrade  of  ranges; 

•  Also  reporting  directly  to  the  CNO,  the 
President  of  the  Board  of  Inspection  and 
Survey  (PRESINSURV)  is  responsible  for 
conducting  acceptance  trials  of  new  ships 
and  aircraft  acquisitions  and  is  the  primary 
Navy  authority  for  production  acceptance 
T&E  of  these  systems; 

•  Conducting  OT&E  on  aviation  sys¬ 
tems  in  conjunction  with  Marine  Corps 
Operational  Test  amd  Evaluation  Activity 
(MCOTEA). 

3.4.3  Air  Force  Organizational 
Relationships 

3.4.3.1  Air  Force  Acquisition  Executive 

The  Assistant  Secretary  of  the  Air  Force  for 
Acquisition  (ASAF/A)  is  the  senior-level 
authority  for  research,  development  and 
acquisition  within  the  Air  Force.  As  illus¬ 
trated  in  Figure  3-4,  he  is  an  advisor  to  the 
Secretary  of  the  Air  Force  and  interfaces 
directly  with  the  DTE  and  DOT&E.  He 
receives  DT&E  and  OT&E  results  as  a  part 
of  the  acquisition  decision  process.  The 
ASAF/A  has,  within  his  structure,  a  mili¬ 
tary  deputy  (acquisition)  who  is  the  Air 
Force  primary  staff  officer  with  responsi¬ 
bility  for  R&D  and  acquisition.  He  is  the 
chief  advocate  of  Air  Force  acquisition  pro¬ 
grams  and  develops  the  RDT&E  budget. 
Air  Force  policy  and  oversight  for  T&E  is 
provided  by  a  staff  element  under  the  Chief 
of  Staff,  Test  and  Evaluation  ( AF  /TE) .  They 
process  test  documentation  for  DT&E  and 
OT&E  and  manage  the  review  of  the  TEMP. 


3.4.3.2  Air  Force  DT&E  Organization 

The  Air  Force  Materiel  Command  (AFMC) 
is  the  primary  DT&E  and  acquisition  man¬ 
ager.  The  AFMC  performs  all  levels  of  re¬ 
search;  develops  weapons  systems,  sup¬ 
port  systems  and  equipment;  and  conducts 
all  DT&E.  The  acquisition  program  manag¬ 
ers  are  under  the  Commander,  AFMC. 
Within  the  AFMC,  there  are  major  product 
divisions,  test  centers  and  laboratories  as 
well  as  missile,  aircraft  and  munitions  test 
ranges. 

Once  the  weapon  system  is  fielded,  AMC 
retains  management  responsibility  for  de¬ 
veloping  and  testing  system  improvements, 
enhancements  or  upgrades. 

3.4.3.3  Air  Force  OT&E  Organizations 

The  AF/TE  is  responsible  for  supporting 
and  coordinating  the  OT &E  activities  of  the 
Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC). 

The  Commander,  Air  Force  Operational 
Test  and  Evaluation  Center,  is  responsible 
to  the  Secretary  of  the  Air  Force  and  the 
Chief  of  Staff  for  the  independent  test  and 
evaluation  of  all  major  and  nonmajor  sys¬ 
tems  acquisitions.  He  is  supported  by  the 
operational  commands  and  others  in  plan¬ 
ning  and  conducting  OT&E. 

The  AFOTEC  develops  operational  require¬ 
ments,  employment  concepts,  tactics,  main¬ 
tenance  concepts,  training  requirements 
and  conducts  OT&E.  The  operational  com¬ 
mands  provide  operational  concepts,  per¬ 
sonnel  and  resources  to  assist  AFOTEC  in 
performing  OT&E. 

3.4.4  Marine  Corps  Organizational 
Relationship 

3.4.4.1  Marine  Corps  Acquisition 
Executive 
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Figure  3-4.  Air  Force  T&E  Organization 


The  Deputy  Chief  of  Staff  for  Research  and 
Development  (DCS/R&D),  Headquarters 
Marine  Corps,  directs  the  total  Marine 
Corps  RDT&E  effort  to  support  the  acqui¬ 
sition  of  new  systems.  His  position  within 
the  General  Staff  is  analogous  to  that  of  the 
Director,  T&E,Tech/N-91  in  the  Navy  struc¬ 
ture.  The  DCS/R&D  also  reports  directly  to 
the  Assistant  Secretary  of  the  Navy/ Re¬ 
search,  Engineering  and  Science  (ASN/ 
RE&S)  in  the  Navy  Secretariat.  Figure  3-3, 
illustrates  the  Marine  Corps  organization 
for  T&E  management. 

3.4.4.2  Marine  Corps  DT&E 
Organizations 

The  Commanding  General,  Marine  Corps 
Systems  Command  (CG  MCSC),  is  the  Ma¬ 
rine  Corps  materiel  developing  agent  di¬ 
rectly  interfaces  with  the  Navy  Systems 
Commands.  The  CG  MCSC  implements 
policies,  procedures  and  requirements  for 
DT&E  of  all  systems  acquired  by  the  Ma¬ 
rine  Corps.  TTie  Marine  Corps  also  uses 
DT&E  and  OT&E  performed  by  other  Ser¬ 
vices,  which  may  develop  systems  of  inter¬ 
est  to  the  Corps. 

3.4.4.3  Marine  Corps  Operational 
Test  and  Evaluation  Agency 

The  MCOTEA  is  the  independent  OT&E 
activity  maintained  by  the  Marine  Corps. 


Its  function  is  analogous  to  that  performed 
by  OPTEVFOR  in  the  Navy.  The  CG  MCSC 
provides  direct  assistance  to  MCOTEA  in 
the  planning,  conduct  and  reporting  of 
OT&E.  The  Fleet  Marine  Force  performs 
troop  test  and  evaluation  of  materiel  devel¬ 
opment  in  an  operational  environment. 

3.5  SUMMARY 

An  increased  emphasis  on  test  and  evalua¬ 
tion  has  placed  greater  demands  on  the 
OSD  and  DOD  components  to  carefully 
structure  organizations  and  resources  to 
ensure  maximum  effectiveness.  Renewed 
interest  by  the  Congress  on  testing  as  a  way 
of  assessing  systems  utility  and  effective¬ 
ness  and  a  recent  report  by  the  President's 
Blue  Ribbon  Panel  on  Acquisition  Manage¬ 
ment  have  resulted  in  major  reorganiza¬ 
tions  within  the  Services.  These  reorgani¬ 
zations  will  be  ongoing  for  several  years  to 
improve  the  program  management  and  test 
and  evaluation  of  acquisition  systems. 
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4 

PROGRAM  OFFICE  RESPONSIBILITIES 
FOR  TEST  AND  EVALUATION 


4.1  INTRODUCTION 

In  government  program  management  of¬ 
fices  (PMOs),  there  should  be  an  element 
dedicated  to  management  of  test  and  evalu¬ 
ation.  This  element  would  have  the  overall 
test  program  responsibility  for  all  phases  of 
the  acquisition  process.  In  the  PMO,  the 
Deputy  for  Test  and  Evaluation  (T&E) 
would  be  responsible  for  defining  the  scope 
and  concept  of  the  test  program,  establish¬ 
ing  the  overall  program  test  objectives  and 
managing  test  program  funds  and  coordi¬ 
nation.  The  Deputy  for  T&E  should  pro¬ 
vide  test  directors  (such  as  a  joint  test  direc¬ 
tor)  as  required,  and  coordinate  the  test 
resources,  facilities  and  their  support  re¬ 
quired  for  each  phase  of  testing.  In  addi¬ 
tion,  he  or  a  member  of  his  staff,  will  be 
responsible  for  managing  the  Test  and 
Evaluation  Master  Plan  (TEMP)  and  plan¬ 
ning  and  managing  the  special  test  pro¬ 
grams  required  for  the  program.  The 
Deputy  for  T&E  will  also  review,  evaluate, 
approve  and  release  for  distribution  con¬ 
tractor-prepared  test  plans  and  reports  and 
review  and  coordinate  all  appropriate  gov¬ 
ernment  test  plans.  After  the  system  is  pro¬ 
duced,  he  will  be  responsible  for  support¬ 
ing  production  acceptance  testing  and  the 
test  portions  of  preplanned  product  im¬ 
provement  (P^I)  upgrades  or  enhance¬ 
ments  to  the  weapon  system/acquisition. 
If  the  program  is  large  enough,  the  Deputy 


for  T&E  will  be  responsible  for  all  T&E 
direction  and  guidance  for  that  program. 

4.2  RELATIONSHIP  TO 
THE  PROGRAM  MANAGER 

The  program  manager  (PM)  is  ultimately 
responsible  for  all  aspects  of  the  system 
development,  including  testing.  The 
Deputy  for  T&E  is  normally  authorized  by 
the  PM  to  conduct  all  duties  in  the  area  of 
test  and  evaluahon.  The  input  of  the  Deputy 
for  T&E  to  the  contract,  engineering  speci¬ 
fications,  budget,  program  schedule,  etc.,  is 
essential  for  the  PM  to  manage  the  pro¬ 
gram  efficiently. 

4.3  EARLY  PROGRAM  STAGES 

In  the  early  stages  of  the  program,  the 
Deputy  for  T&E  writes  the  test  sections  of 
the  Request  for  Proposal  (REP).  Although 
the  ultimate  responsibility  for  the  RFP  is 
between  the  PM  and  the  primary  contract¬ 
ing  officer  (PCO),  the  Deputy  for  T&E  is 
responsible  for  creating  several  sections. 
These  sections  include  the  test  schedule, 
test  program  funding  (projections),  test  data 
requirements  for  the  program  (test  reports, 
plans,  procedures,  quick-look  reports,  etc.), 
the  test  section  of  the  Statement  of  Work 
(SOW),  the  Acquisition  Plan,  Information 
for  Proposal  Preparation  (IFPP),  and  (if  a 
joint  acquisition  program)  the  Joint  Opera¬ 
tional  Requirements  Document  (JORD). 
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4.3.1  Memorandums 

Early  in  the  program,  another  task  of  the 
Deputy  for  T&E  is  the  arrangement  of  any 
Memorandums  of  Agreement  or  Under¬ 
standing  (MOA/MOU)  between  Services, 
NATO  countries,  test  organizations,  etc., 
which  outline  the  responsibilities  of  each 
organization.  The  RFP  outlines  contractor/ 
government  obligations  and  arrangements 
on  the  access  and  use  of  test  facilities 
(contractor-  and  government-owned). 

4.3.2  Test  Data  Management 

The  Deputy  for  T&E  may  have  approval 
authority  for  all  contractor-created  test 
plans,  procedures  and  reports.  He  must 
have  access  to  all  contractor  testing  and  test 
results,  and  he  is  responsible  for  dissemi¬ 
nating  the  results  to  government  agencies 
that  need  this  data.  Additionally,  the 
Deputy  creates  report  formats  and  time 
lines  for  contractor  submittal,  government 
approval,  etc. 

The  data  requirements  for  the  entire  test 
program  are  outlined  in  the  Contract  Data 
Requirements  List  (CDRL).  The  Deputy  for 
T&E  should  review  the  Acquisition  Man¬ 
agement  Systems  and  Data  Requirements 
Control  List  (AMSDL),  DOD  5010.12-L,  for 
relevant  test  data  item  descriptions  (DIDs). 
(Examples  can  be  found  in  Appendix  C.) 
The  Deputy  for  T&E  provides  input  to  this 
section  of  the  RFP  early  in  the  program.  He 
ensures  that  his  office  and  all  associated 
test  organizations  requiring  the  informa¬ 
tion  receive  the  test  documentation  on  time. 
Usually,  the  contractor  sends  the  data  pack¬ 
ages  directly  to  the  Deputy  for  T&E,  who, 
in  turn,  has  a  distribution  list  trimmed  to 
the  minimum  number  of  copies  for  agen¬ 
cies  needing  that  information  to  perform 
their  mission  and  oversight  responsibili¬ 
ties.  It  is  important  for  the  Deputy  for  T&E 


to  use  an  integrated  test  program  and  re¬ 
quest  contractor  test  plans  and  procedures 
well  in  advance  of  the  actual  tests  perfor¬ 
mance  to  ensure  his  office  has  time  to  ap¬ 
prove  the  procedures  or  implement  modi¬ 
fications.  Conversely,  he  must  receive  the 
test  results  and  reports  on  time  to  enable 
him,  the  PM  and  higher  authorities  to  make 
program  decisions.  Further,  the  data  re¬ 
ceived  should  be  tailored  to  provide  the 
minimum  information  and  copies  needed. 
The  Deputy  for  T&E  must  be  aware  that 
data  requirements  in  excess  of  the  mini¬ 
mum  needed  will  lead  to  an  unacceptable 
increase  in  overall  program  cost.  For  data 
that  is  needed  quickly  and  informally  (at 
least  initially),  the  Deputy  for  T&E  can 
request  Quick-Look  Reports  that  give  test 
results  immediately  after  test  performance. 
The  Deputy  for  T&E  is  also  responsible  for 
coordinating  with  the  contractor  on  all  re¬ 
port  formats  (the  in-house  contractor  for¬ 
mat  is  acceptable  in  most  cases). 

4.3.3  Test  Schedule  Formulation 

A  very  important  task  the  Deputy  for  T&E 
has  during  the  creation  of  the  RFP  is  the  test 
program  schedule.  Initially,  the  PM  will 
need  contractor  predictions  of  the  hard¬ 
ware  (and  software  in  some  cases)  avail¬ 
ability  dates  for  models,  prototypes, 
mockups,  full-scale  models,  etc.,  once  the 
contract  is  awarded.  The  Deputy  for  T&E 
uses  this  information  to  create  a  realistic 
front-end  schedule  of  the  in-house  testing 
the  contractor  will  conduct  before  govern¬ 
ment  testing  (development  testing  (DT)  and 
operational  testing  (OT)).  Then,  a 
"strawman"  schedule  is  developed  upon 
which  the  government  DT  and  OT  sched¬ 
ules  can  be  formulated  and  contractor  sup¬ 
port  requirements  determined.  The  Deputy 
for  T&E  can  use  past  experience  in  testing 
similar  weapon  systems/acquisition  items 
or  contract  test  organizations  that  have  the 
required  experience  to  complete  the  entire 
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test  schedule.  Since  the  test  schedule  is  a 
critical  contractual  item,  contractor  input  is 
very  important.  The  test  schedule  will  nor¬ 
mally  become  an  item  for  negotiation  once 
the  RFP  is  released  and  the  contractor's 
proposal  is  received.  Attention  must  be 
given  to  ensuring  the  test  schedule  is  not 
too  success-oriented  so  test  failures  will  not 
result  in  serious  program  delays  for  either 
the  government  test  agencies  or  the  con¬ 
tractor. 

4.3.4  Programmatic  Environment 
Analysis 

The  PMO  personnel  should  be  sensitive  to 
the  potential  environmental  consequences 
of  system  materials,  operations  and  dis¬ 
posal  requirements.  Public  laws  (Title  40, 
Code  of  Federal  Regulations,  Parts  1500- 
1508;  National  Environmental  Policy  Act 
(NEPA)  Regulations;  Executive  Order 
12114,  Environinental  Effects  Abroad  of 
Major  Federal  Actions;  DOD  Instruction 
5000.2,  part  3;  etc.)  require  analysis  of  haz¬ 
ardous  materials  and  appropriate  mitiga¬ 
tion  measures  during  each  acquisition 
phase.  As  stated  in  DOD  Instruction  5000.2, 
part  6-1,  "Emphasis  shall  be  on  reduced  use 
of  hazardous  materials  in  processes  and 
products  rather  than  simply  managing  the 
hazardous  waste  created." 

Litigations  resulting  in  personal  fines  and 
imprisonment  successfully  executed 
against  government  employees  have  raised 
the  environmental  awareness  at  test  ranges 
and  facilities.  Environmental  Impact  State¬ 
ments  (supported  by  long,  thorough  stud¬ 
ies  and  public  testimony)  or  Environmen¬ 
tal  Analysis  and  Assessments  (DOD 5000.2- 
M,  4-F)  are  generally  required  before  any 
system  testing  can  be  initiated.  A  summary 
of  the  environmental  analysis  appears  in 
the  Integrated  Program  Summary  (IPS)  and 
is  updated  for  each  milestone  decision  point. 


4.4  PMO/CONTRACTOR 
TEST  MANAGEMENT 

The  PMO  will,  in  most  cases,  have  a  con¬ 
tractor  test  section  counterpart.  With  this 
counterpart,  the  Deputy  for  T&E  works 
out  the  detailed  test  planning,  creation  of 
schedules,  etc.,  for  the  entire  test  program. 
1  he  PMO  uses  input  from  ail  sources  (con¬ 
tracts,  development  test  agencies,  opera¬ 
tional  test  agencies,  higher  headquarters, 
etc.)  to  formulate  the  test  program's  length, 
scope  and  necessary  details.  The  Deputy 
for  T&E  ensures  that  the  RFP  reflects  the 
test  program  envisioned  and  the 
contractor's  role  in  the  acquisition.  He  also 
ensures  the  RFP  includes  provisions  for 
government  attendance  at  contractor's  tc^ts 
and  that  all  contractor  test  results  are  pro¬ 
vided  to  the  government. 

After  the  RFP  has  been  issued  and  the 
contractor  has  responded,  the  proposal  is 
reviewed  by  the  PMO.  The  Deputy  for  T&E 
is  responsible  for  performing  a  technical 
evaluation  on  the  test  portions  of  the  pro¬ 
posal.  In  this  technical  evaluation,  he  com¬ 
pares  the  proposal  to  the  SOW,  test  sched¬ 
ule,  IFPP,  etc.,  and  reviews  the  contractor's 
cost  of  each  testing  item.  This  is  an  iterative 
process  of  refining,  clarifying  and  modify¬ 
ing  that  will  ensure  the  final  contract  be¬ 
tween  the  PMO  and  the  prime  contractor 
(subcontractors)  contains  all  test-related 
tasks  and  is  priced  within  scope  of  the 
proposed  test  program.  Once  technical 
agreement  on  the  contractor's  technical 
approach  is  reached,  the  Deputy  for  T&E  is 
responsible  for  giving  inputs  to  the  govern¬ 
ment  contracting  officer  during  contract 
negotiations.  The  contracting  officer-re¬ 
quested  contract  deliverables  are  assigned 
contract  line  item  numbers  (CLINs),  which 
are  created  by  the  Deputy  for  T&E.  This 
will  ensure  the  contractor  delivers  the  re¬ 
quired  performances  at  specified  intervals 
during  the  life  of  the  contract.  Usually, 
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there  will  be  separate  contracts  for  devel¬ 
opment  and  production  of  the  acquisition 
item.  For  each  type  of  contract,  the  Deputy 
for  T&E  has  the  responsibility  to  provide 
the  PCO  and  PM  with  the  test  and  evalua¬ 
tion  input. 

4.5  TEST  PLANNING 
WORKING  GROUPS 

Before  the  final  version  of  the  RFP  is  cre¬ 
ated,  the  Deputy  for  T&E  will  form  a  test 
planning  working  group.  This  group  in¬ 
cludes  the  operational  test  agency,  devel¬ 
opment  test  agency,  organizations  that  may 
be  jointly  acquiring  the  same  system,  the 
test  supporting  agencies,  operational  us¬ 
ers,  and  any  other  organizations  that  will 
be  involved  in  the  test  program  by  provid¬ 
ing  test  support  or  by  conducting,  evaluat¬ 
ing  or  reporting  on  testing.  In  later  meet¬ 
ings,  the  contractor  participates  in  this  test 
planning  group;  however,  the  contractor 
iitay  not  be  selected  by  the  time  the  first 
meetings  are  held. 

The  purposes  of  these  meetings  are  to  re¬ 
view  and  assist  in  the  development  of  early 
test  documentation,  the  TEMP,  and  to  agree 
on  basic  test  program  schedules,  scope, 
support,  etc.  TTie  TEMP  serves  as  the  top- 
level  test  management  document  for  the 
acquisition  program,  being  updated  as  the 
changing  program  dictates. 

4.6  TEST  PROGRAM  FUNDING/ 
BUDGETING 

The  PMO  must  identify  funds  for  testing 
very  early  so  that  test  resources  can  be 
obtained.  The  Deputy  for  T&E  uses  the 
acquisition  schedule,  TEMP  and  other  pro¬ 
gram  and  test  documentation  to  identify 
test  resource  requirements.  He  coordinates 
these  requirements  with  the  contractor  and 
government  organizations  that  have  the 
test  facilities  to  ensure  their  availability  for 


testing.  The  Deputy  for  T&E  ensures  that 
test  costs  include  contractor  and  govern¬ 
ment  test  costs.  The  contractor's  test  costs 
are  normally  outlined  adequately  in  his 
proposal;  however,  the  government  test 
ranges,  instrumentation  and  test-support 
resource  costs  must  be  determined  by  other 
means.  Usually,  the  Deputy  for  T&E  con¬ 
tacts  the  test  organization  and  outlines  his 
test  program  requirements  (Uniform  Docu¬ 
ment  System);  and  the  test  organization 
sends  the  program  office  an  estimate  of  the 
test  program  costs.  He  then  obtains  cost 
estimates  from  all  test  sources  he  antici¬ 
pates  using  and  supplies  this  information 
to  the  PM  The  Deputy  for  T&E  must  also 
ensure  that  any  program  funding  reduc¬ 
tions  are  not  absorbed  entirely  by  the  test 
program.  Some  cutbacks  may  be  necessary 
and  allowable;  but  the  test  program  must 
supply  the  PM,  other  defense  decision¬ 
making  authorities,  and  the  Congress  with 
enough  information  to  make  prograiii  mile¬ 
stone  decisions. 

4.7  TECHNICAL  REVIEWS, 

DESIGN  REVIEWS  AND  AUDITS 

The  role  of  the  Deputy  for  T&E  changes 
slightly  during  the  contractor's  technical 
reviews,  design  reviews,  physical  and  func¬ 
tional  configuration  audits,  etc.  Usually  he 
plans,  directs  or  monitors  government  test¬ 
ing;  however,  in  the  reviews  and  audits,  he 
examines  the  contractor's  approach  to  the 
test  problem  and  evaluates  the  validity  of 
the  process  and  the  accuracy  of  the 
contractor's  results.  Using  his  experience 
and  background  in  test  and  evaluation,  he 
assesses  whether  the  contractor  did  enough 
or  too  little  testing;  whether  the  tests  were 
biased  in  any  way;  and  if  they  followed  a 
logical  progression  using  the  minimum  of 
time,  effort  and  funds.  If  the  Deputy  for 
T&E  finds  any  discrepancies,  he  must  in¬ 
form  the  contractor,  the  PM  and  the  PCO  to 
validate  his  conclusions  before  effecting 
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corrections.  Each  type  of  review  or  audit 
will  have  a  different  focus /orientation,  but 
the  Deputy  for  T&E  will  always  be  con¬ 
cerned  with  the  testing  process  and  how  it 
is  carried  out.  After  each  review,  the  Deputy 
for  T&E  should  always  document  his  ob¬ 
servations  for  future  reference. 

4.8  CONTRACTOR  TESTING 

The  Deputy  for  T&E  is  responsible  for  en¬ 
suring  that  contractor-conducted  tests  are 
monitored.  He  must  also  be  given  access  to 
all  contractor  internal  data,  test  results  and 
test  reports  related  to  his  acquisition  pro¬ 
gram.  Usually,  the  contract  requires  that 
government  representatives  be  informed 
ahead  of  time  of  any  (significant  or  other¬ 
wise)  testing  the  contractor  conducts  so  the 
government  can  arrange  to  witness  certain 
testing  or  receive  results  of  the  tests.  Fur¬ 
ther,  the  contractor's  internal  data  should 
be  available  as  a  contract  provision.  The 
Deputy  for  T&E  must  ensure  that  govern¬ 
ment  test  personnel  (development  test  and 
evaluation /operational  test  and  evaluation) 
have  access  to  contractor  test  results.  It 
would  be  desirable  to  have  all  testers  ob¬ 
serve  some  contractor  tests  to  help  develop 
confidence  in  the  results  and  identify  areas 
of  risk. 

4.9  SPECIFICATIONS 

Within  the  program  office,  the  engineering 
section  is  usually  tasked  to  create  the  pre¬ 
liminary  specifications  for  release  of  the 
RFP.  The  contractor  is  then  tasked  with 
creating  the  specification  documentation 
called  out  by  the  contract,  which  will  be 
delivered  once  the  item /system  design  is 
formalized  for  production.  The  Deputy  for 
T&E  performs  an  important  function  in 
specification  formulation  by  reviewing  the 
specifications  to  determine  if  performance 
parameters  are  testable;  if  current,  state-of- 
the-art  technology  can  determine  (during 


the  DT&E  test  phase)  if  the  performance 
specifications  are  being  met  by  the  acquisi¬ 
tion  item;  or  if  the  specified  parameters  are 
too  "tight."  A  specification  is  too  "tight"  if 
the  requirements  are  impossible  to  meet  or 
demonstrate,  or  if  the  specification  has  no 
impact  on  the  form,  fit  or  function  of  the 
end-item,  the  system  it  will  become  a  part 
of  or  the  system  with  which  it  will  interact. 
He  must  determine  if  test  objectives  can  be 
adequately  formulated  from  those  specifi¬ 
cations  that  will  provide  thresholds  of  per¬ 
formance,  minimum  and  maximum  stan¬ 
dards  and  reasonable  operating  conditions 
for  the  end-item's  final  mission  and  operat¬ 
ing  environment.  The  specifications  shape 
the  development  test  and  evaluation 
(DT&E)  testing  scenario,  test  ranges,  test 
support,  targets,  etc.,  and  are  very  impor¬ 
tant  to  the  Deputy  for  T&E. 

4.10  INDEPENDENT 
EVALUATION  AGENCIES 

The  PMO  Deputy  for  T&E  does  not  have 
direct  control  over  government-owned  test 
resources,  lest  facilities,  test  ranges,  test 
personnel,  etc.  Therefore,  he  must  depeiid 
on  those  test  organizatioi  controlling  them 
and  stay  involved  with  the  test  agency 
activities.  The  amount  of  involvement  de¬ 
pends  on  the  item  being  tested;  its  com¬ 
plexity,  cost  and  characteristics;  the  length 
of  time  for  testing;  amount  of  test  funds; 
etc.  Usually,  the  "nuts  and  bolts"  detailed 
test  plans  and  procedures  are  written  by 
the  test  organizations  controlling  the  test 
resources  with  input  and  guidance  from 
the  Program  Office  Deputy  for  T&E.  The 
Deputy  for  T&E  is  responsible  for  ensuring 
that  the  tests  are  performed  using  test  ob¬ 
jectives  based  on  the  specifications  and  that 
the  requirements  of  timeliness,  accuracy 
and  minimal  costs  are  met  by  the  test  pro¬ 
gram  design.  During  the  testing,  the  Deputy 
for  T&E  monitors  test  results.  The  test 
agencies  submit  a  copy  of  their  report  to  the 
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Program  Office  at  the  end  of  testing,  usu¬ 
ally  to  the  Office  of  the  Deputy  for  T&E. 

4.11  SUMMARY 

Staffing  requirements  in  the  PMO  vary  with 
the  program  phase  and  the  T&E  workload. 
Test  and  evaluation  expertise  is  essential  in 
the  early  planning  stages  but  can  be  pro¬ 
vided  through  matrix  support.  The  Deputy 


for  T&E  may  be  subordinate  to  the  chief 
engineer  in  early  phases  but  should  be¬ 
come  a  separate  staff  element  after  Mile¬ 
stone  (MS)  II.  Changing  of  critical  players 
can  destroy  established  working  relation¬ 
ships  and  abrogate  prior  agreements  if  con¬ 
tinuity  is  not  maintained.  The  PMO  man¬ 
agement  of  T&E  must  provide  for  an  inte¬ 
grated  focus  and  a  smooth  transition  from 
one  staff-support  mode  to  the  next. 
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TEST-RELATED 

DOCUMENTATION 


5.1  INTRODUCTION 

During  the  course  of  a  defense  acquisition 
program,  many  documents  are  developed 
that  have  significance  for  those  responsible 
for  testing  and  evaluating  the  system.  This 
chapter  is  designed  to  provide  background 
on  some  of  these  documents. 

As  Figure  5-1  shows,  test-related  documen¬ 
tation  spans  a  broad  range  of  materials.  It 
includes  requirements  documentation  such 
as  the  Mission  Need  Statement  (MNS);  pro¬ 
gram  decision  documentation  such  as  the 
Integrated  Program  Summary  (IPS)  and 
Acquisition  Decision  Memorandum 
(ADM);  and  program  management  docu¬ 
mentation  such  as  the  Acquisition  Strat¬ 
egy,  Baseline  documentation,  the  System 
l^gineering  Management  Plan  (SEMP),  the 
Integrated  Logistics  Support  Plan  (ILSP) 
and  the  Test  and  Evaluation  Master  Plan 
(TEMP).  Of  importance  to  the  PM  and  to 
test  and  evaluation  (T&E)  managers  are 
additional  test  program  documents  such  as 
specific  test  designs,  test  plans,  outline  test 
plans/ test  program  outlines,  evaluation 
plans  and  test  reports.  This  chapter  con¬ 
cludes  with  a  description  of  the  End-of- 
Test  Phase  and  beyond  Low-Rate  Initial 
Production  (LRIP)  Reports,  two  special- 
purpose  T&E  status  reports  ti\at  are  used  to 
support  the  milestone  decision  process. 

5.2  REQUIREMENTS 
DOCUMENTATION 

5.2.1  Continuing  Mission  Area  Analyses 

As  indicated  in  DODD  5000.1,  the  Services 
are  required  to  conduct  continuing  mission 


analyses  of  their  assigned  areas  of  respon¬ 
sibility.  These  Mission  Area  Analyses 
(MAA)  may  result  in  recommendations  to 
initiate  new  acquisition  programs  to  re¬ 
duce  or  eliminate  operational  defic’encies. 
If  a  need  cannot  be  met  through  changes  in 
tactics,  strategy,  doctrine  or  training  and  a 
materiel  solution  is  required,  the  needed 
capability  is  described  in  a  document 
known  as  an  Operational  Requirement 
Etocument  (ORD).  When  the  cost  of  a  pro¬ 
posed  acquisition  program  is  estimated  to 
exceed  $200  million  for  research,  develop¬ 
ment,  test  and  evaluation  or  $1  billion  for 
procurement  (FY 1980$),  it  is  considered  a 
major  program  and  requires  an  MNS.  The 
MAA  is  completed  at  the  beginning  of  a 
program  and  reviewed  to  evaluate  system 
modifications  periodically. 

5.2.2  Mission  Need  Statement  (MNS) 

The  MNS  is  a  short,  nonsystem-specific 
statement  of  operational  capability  need 
prepared  by  any  DOD  component  focus¬ 
ing  on  a  specific  mission  area  need  or  defi¬ 
ciency.  Service  validation  and,  for  those 
potential  Acquisition  Category  (ACAT)  I 
Programs,  review  and  validation  by  the 
Joint  Requirements  Oversight  Council 
QRCXr)  results  in  forwarding  of  the  MNS  to 
^e  milestone  (MS)  decision  authority  for 
MS  0  consideration.  The  five-page 
document's  content  and  format,  as  de¬ 
scribed  in  PT  2,  DOD  50()0.2-M,  includes: 

•  Identification  of  the  applicable  Defense 
Plaiming  Guidance  Element; 
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Figure  5-1 .  Test-Related  Documentation 


•  Mission  and  threat  analyses  —  need 
defined  in  terms  of  mission,  objectives  and 
general  capabilities; 

•  Nonmateriel  alternatives —  tactics,  doc¬ 
trine,  organization  and  training; 

•  Potential  materiel  alternatives  —  NDI, 
allied,  inter-Service,  and  new; 

•  Constraints  by  infrastructure,  treaties 
and  environments. 

The  MNS  and  other  requirements  docu¬ 
ments  are  of  particular  value  to  the  tester 
since  they  form  the  basis  for  the  initial 
identification  of  critical  issues  that  will  be 
addressed  in  the  test  program. 

5.2.3  Operational  Requirements 
Document  (ORD) 

The  ORD  is  first  prepared  for  MS  I  by  the 
user  or  a  user's  representative  and  is  ap¬ 
proved  by  the  Service  Chief  or  a  designated 
representative.  At  MS  II,  the  updated  ORD 
should  contain  "thresholds  and  objectives 
for  more  detailed  and  refined  performance 
capabilities  and  characteristics  based  on 
the  results  of  trade-off  studies  and  testing 
conducted  during  Phase  I,  Demonstration 
and  Validation."  The  ORD  is  a  translation 
of  the  MNS  into  user  requirements  and 
each  concept  considered  at  MS  1  will  have 
a  tailored  ORD.  Objectives  and  thresholds 
for  various  system  performance  param¬ 
eters  outlined  in  the  ORD  will  also  be  found 
in  baseline  documents,  the  TEMP  and  pro¬ 
gram  specifications. 

5.2.4  System  Threat  Assessment 
Report  (STAR) 

A  STAR  is  prepared  by  the  DOD  Compo¬ 
nent  Intelligence  Command  or  Agency,  and 
AC  AT  ID  STARs  are  validated  by  the  De¬ 
fense  Intelligence  Agency.  The  STAR  will 


contain  a  "concise  description  of  the  pro¬ 
jected  future  operational  threat  environ¬ 
ment,  the  system-specific  threat,  the  reac¬ 
tive  threat  that  could  affect  program  deci¬ 
sions,  and  when  appropriate,  the  results  of 
interactive  analysis  obtained  by  the  Service 
Program  Manager  when  evaluating  the 
program  against  the  threat."  Threat  projec¬ 
tions  start  at  the  initial  operating  capacity 
(IOC)  and  extend  over  the  system's  ex¬ 
pected  operational  life.  The  STAR  provides 
the  basis  for  the  test  design  of  threat  sce¬ 
narios  and  the  acquisition  of  appropriate 
threat  equipment  or  surrogates.  It  provides 
threat  data  for  development  test  and  evalu¬ 
ation  (DT&E)  and  operational  test  and 
evaluation  (OT&E).  Vulnerability  and  le¬ 
thality  analyses  during  live  fire  testing  of 
ACAT  I  and  II  systems  are  contingent  on 
valid  threat  descriptions.  A  summary  of 
the  STAR  are  included  in  the  TEMP. 

5.3  PROGRAM  DECISION 
DOCUMENTATION 

5.3.1  Acquisition  Decision 
Memorandum  (ADM) 

Under  Secretary  of  Defense  for  Acquisition 
(USD(A))  decisions  at  major  acquisition 
program  (ACAT  ID)  milestones  are  re¬ 
corded  in  a  document  known  as  an  Acqui¬ 
sition  Decision  Memorandum  (ADM).  The 
ADM  documents  a  USD(A)  decision  on  an 
MNS  at  MS  0  and  on  the  IPS  at  MSs  I,  II  and 
III.  In  conjunction  with  an  ADM  and  its 
included  exit  criteria  for  the  next  phase,  the 
IPS  is  a  primary  program  guidance  docu¬ 
ment  providing  goals /thresholds  for  sys¬ 
tems  performance.  (PT  13,  DODI  5000.2) 

5.3.2  Integrated  Program 
Summary  (IPS) 
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The  IPS,  prepared  by  the  program  execu¬ 
tive  officer  (PEO)  with  PM  support,  pro¬ 
vides  a  comprehensive  summary  of  pro- 
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Figure  5-2.  Requirements  Definition  Process 


gram  structure,  status,  assessment,  plans 
and  recommendations  by  the  PM  and  the 
PEO.  It  addresses  current  program  status, 
future  plans  and  RISK  areas.  It  provides  for 
the  establishment  of  explicit  program  cost, 
schedule  and  performance  objectives, 
thresholds  included  in  the  acquisition  pro¬ 
gram  baseline  and  the  next  phase's  pro¬ 
posed  exit  criteria.  At  MS  II,  the  IPS  pro¬ 
vides  information  on  LRIP  quantities  (total 
production,  initial  operational  test  and 
evaluation  (lOT &E)  assets)  along  with  T &E 
events  leading  up  to  the  start  of  LRIP.  The 
format  and  content  of  the  IPS  are  provided 
in  PT  4,  DOD  5000.2-M.  Test  managers  will 
find  that  IPS  information  helps  in  defining 
program  scope  and  identifies  key  areas  of 
technological  RISK  that  will  influence  the 
test  program  structure  and  duration.  Test 
planning  documents,  especially  the  TEMP, 
should  be  compatible  with  the  annexes  to 
the  IPS. 

Annex  A — Program  structure  vs.  the  TEMP 
integrated  program  summary,  PT  II 

Annex  B — Program  life-cycle  cost-estimate 
summary  vs.  funding  profile  TEMP,  PT  II 
and  PT  V 

Annex  C — Acquisition  Strategy  Report  vs. 
PT  III,  DT&E  and  PT  IV,  OT&E 

Annex  D — RISK  Assessment  vs.  PT  III, 
DT&E 

Annex  E — Environmental  Analysis  vs.  PT 
III,  DT&E  and  PT  IV,  OT&E. 

5.3.3  Cost  and  Operational 
Effectiveness  Analysis  (COEA) 

A  COEA  is  normally  prepared  by  a  Service 
agency  other  than  the  program  manage¬ 
ment  office.  The  COEA  aids  decision-mak¬ 
ers  by  examining  the  relative  advantages 
and  disadvantages  of  program  alternatives, 
providing  the  rationales  for  each  option. 


The  guidance  in  DODI  5000.2,  part  4-E,  was 
tied  more  directly  to  test  and  evaluation 
measures  of  effectiveness  (MOE)  by  a  memo 
issued  by  the  USD(A)  in  March  1992  and 
cosigned  by  the  Assistant  Secretary  of  De¬ 
fense  for  Program  Analysis  and  Evaluation 
(ASD(PAE))  and  the  Director,  Operational 
Test  and  Evaluation  (DOT&E).  It  stated: 

The  DOD  component  in  the  process  of  per¬ 
forming  a  MSI  COEA,  should  identify  the 
MOE's  [Measures  of  Effectiveness]  to  be 
used  in  the  COEA  and  show  how  these 
MOE's  are  derived  from  the  MNS.  Each 
COEA  should  include  MOE's  reflecting 
operational  utility  that  can  be  tested. 

The  TEMP  should  document  how  the 
COEA  MOE's  and  related  measures  of 
performance  (MOP)  will  be  addressed 
in  test  and  evaluation. 

In  particular,  the  MOE's,  MOP's  and 
criteria  in  the  ORD,  the  COEA,  the 
TEMP  and  the  APB,  should  be  consis¬ 
tent. 

In  assessing  the  possible  impact  of  test 
limitations,  the  DOD  component  re¬ 
sponsible  for  the  COEA  should  ex¬ 
plain  in  a  quantitative  evaluation  how 
and  to  what  extent  COEA  results 
would  be  expected  to  vary  as  a  result 
of  test  limitations. 

The  driving  factor  behind  this  linkage  is  the 
decision-maker's  reluctance  to  accept  mod¬ 
eling  or  simulation  projections  for  system 
performance  in  the  future  without  actual 
test  data  that  validates  COEA  results. 

5.4  PROGRAM  MANAGEMENT 
DOCUMENTATION 

5.4.1  Acquisition  Strategy 

An  event-based  acquisition  strategy  must 
be  formulated  at  the  start  of  a  development 
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program  (MS  I).  Event-driven  acquisition 
strategy  explicitly  links  program  decisions 
to  demonstrated  accomplishments  in  de¬ 
velopment,  testing  and  initial  production. 
The  strategy  constitutes  a  broad  set  of  con¬ 
cepts  that  provide  direction  and  control  for 
the  overall  development  and  production 
effort.  The  acquisition  strategy  is  updated 
at  each  MS  decision  point  throughout  the 
life  of  a  program.  The  level  of  detail  re¬ 
flected  in  the  acquisition  strategy  can  be 
expected  to  increase  as  a  program  matures. 
The  acquisition  strategy  serves  as  a  concep¬ 
tual  basis  for  formulating  functional  plans 
such  as  the  System  Engineering  Manage¬ 
ment  Plan,  Integrated  Logistics  Support 
Plan,  and  the  Test  and  Evaluation  Master 
Plan. 

It  is  important  that  T&E  interests  be  repre¬ 
sented  as  the  acquisition  strategy  is  formu¬ 
lated  because  the  acquisition  strategy 
should: 

•Provide  an  overview  of  the  T&E 
planned  for  the  program,  ensuring  that 
adequate  T&E  is  conducted  prior  to  the 
production  decision; 

•  Discuss  plans  for  providing  adequate 
quantities  of  test  hardware; 

•  Describe  levels  of  concurrence  and  com¬ 
bined  development  test/operational  test 
(DT/OT). 

5.4.2  Baseline  Documentation 

The  acquisition  program  baseline  will  ini¬ 
tially  be  developed  by  the  PM  at  MS  I 
(concept)  and  revised  for  each  subsequent 
milestone  (MS  II — Development,  MS  III — 
Production).  Baseline  parameters  represent 
the  cost,  schedule  and  performance  objec¬ 
tives  and  thresholds  for  the  system  in  a 
production  configuration.  Each  baseline 
influences  the  T&E  activities  in  the  suc¬ 
ceeding  phases.  Guidance  on  the  formula¬ 


tion  of  baselines  is  found  in  PT  11,  EXDDI 
5000.2;  and  the  format  is  in  PT  14,  DOD 
5000.2-M.  A  requirement  that  baselines  in¬ 
corporate  the  contract  specifications  appli¬ 
cable  to  each  baseline  parameter  was  is¬ 
sued  by  the  USD(A)  in  March  1991.  Perfor¬ 
mance  demonstrated  during  T&E  of  pro¬ 
duction  systems  must  meet  or  exceed  the 
thresholds.  The  thresholds  establish  devia¬ 
tion  limits  (actual  or  anticipated  breach 
triggers  reports  —  PT  19,  DOD  5000.2-M) 
for  parameters  beyond  which  the  PM  may 
not  trade  off  cost,  schedule  or  performance 
without  authorization  by  the  MS  decision 
authority.  Baseline  and  test  documentation 
must  reflect  the  same  expectations  for  sys¬ 
tem  performance. 

5.4.3  Integrated  Logist> ;  >  Support  Plan 

Integrated  logistics  support  (ILS)  is  a  com¬ 
posite  of  all  support  considerations  neces¬ 
sary  to  ensure  the  effective  and  economical 
support  of  a  system  at  all  levels  of  mainte¬ 
nance  for  its  programmed  life  cycle  (Refer¬ 
ence  64).  The  ILSP  describes  the  overall  ILS 
program  and  includes  ILS  requirements, 
tasks  and  milestones  for  the  current  and 
succeeding  phases  of  the  program.  The  ILSP 
serves  as  the  source  document  for  ILS  test¬ 
ing  requirements. 

Standards  and  procedures  for  logistic  sup¬ 
port  analysis  (LSA)  are  documented  in  MIL- 
STD-1388-1A.  This  standard  requires  that 
T&E  programs  be  planned  to  serve  the 
following  three  logistics  supportability 
objectives; 

(1)  Provide  measured  data  for  input  into 
system-level  estimates  of  readiness,  opera¬ 
tional  costs  and  logistics  support  resource 
requirements; 

(2)  Expose  supportability  problems  so 
they  can  be  corrected  prior  to  deployment; 
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(3)  Demonstrate  contractor  compliance 
with  quantitative  supportability  —  related 
design  requirements. 

Development  of  an  effective  T&E  program 
requires  close  coordination  of  efforts  among 
all  system  engineering  disciplines,  espe¬ 
cially  those  involved  in  logistics  support 
analyses.  The  ILSP  should  be  drafted  be¬ 
fore  MS  I  to  provide  a  skeletal  framework 
for  logistics  support  analysis,  to  identify 
initial  logistics  testing  requirements  that 
can  be  used  as  input  to  the  TEMP  and  to 
provide  test  feedback  to  support  ILS  devel¬ 
opment. 

5.5  TEST  PROGRAM 
DOCUMENTATION 

5.5.1  Test  and  Evaluation  Master  Plan 

The  Test  and  Evaluation  Master  Plan 
(TEMP)  is  the  basic  planning  document  for 
all  T&E  related  to  a  DOD  system  acquisi¬ 
tion.  It  is  prepared  by  the  program  man¬ 
agement  office  with  the  operational  test 
information  provided  by  the  Service  Op¬ 
erational  Test  Organization.  It  is  used  by 
Office  of  the  Secretary  of  Defense  (OSD) 
and  the  Services  for  planning,  reviewing 
and  approving  T&E  programs  and  pro¬ 
vides  the  basis  and  authority  for  all  other 
detailed  T&E  planning  documents.  The 
TEMP  identifies  critical  technical  charac¬ 
teristics  and  critical  operational  issues 
(COI);  and  it  describes  the  objectives,  re¬ 
sponsibilities,  resources  and  schedules  for 
all  completed  and  planned  T&E.  The  TEMP 
is  required  by  EXDDI  5000.2  (see  Chapter  17 
for  more  information  regarding  the  TEMP); 
guidelines  for  its  preparation  are  found  in 
DOD  5000.2-M,  PT  7. 

5.5.2  Evaluation  Plan 

Evaluation  planning  is  usually  included 
within  the  test  plan.  Evaluation  planning 


considers  the  evaluation  and  analysis  tech¬ 
niques  that  will  be  required  once  the  test 
data  has  been  collected  and  processed. 
Evaluation  is  linked  closely  to  the  test  de¬ 
sign,  especially  the  statistical  models  on 
which  the  test  design  is  built. 

The  Army  requires  the  development  of  a 
TEMP,  with  one  part  dedicated  to  the  evalu¬ 
ation  being  conducted  by  a  technical  inde¬ 
pendent  evaluator  or  an  operational  inde¬ 
pendent  evaluator. 

The  objective  of  the  Army's  emphasis  on 
evaluation  is  to  "address  the  issues;  de¬ 
scribe  the  evaluation  of  issues  which  re¬ 
quire  data  from  sources  other  than  test; 
state  the  technical  or  operational  issues  and 
criteria;  identify  data  sources;  state  the  ap¬ 
proach  to  the  independent  evaluation; 
specify  the  analytical  plan  and  identify  pro¬ 
gram  constraints."  (Reference  54) 

Evaluation  plans  are  prepared  for  all  sys¬ 
tems  in  development  by  the  operational 
evaluators  during  concept  exploration  and 
incoordination  with  the  system  developer. 
The  Army  Master  Evaluation  Plan  becomes 
an  annex  to  the  TEMP  and  is  updated  when 
the  TEMP  is  revised.  It  identifies  each 
evaluation  issue  and  the  methodology  to 
be  used  to  assess  it  and  specifies  require¬ 
ments  for  exchange  of  information  between 
the  development/ operational  testers  and 
materiel  developers. 

5.5.3  Test  Design 

Test  designers  need  to  ensure  that  the  test  is 
constructed  to  provide  useful  information 
in  all  areas/aspects  that  will  lead  to  an 
assessment  of  the  system  performance.  For 
example,  a  complicated,  even  ingenious, 
test  that  does  not  provide  the  information 
required  by  the  decision-makers  is,  in  many 
respects,  a  failed  endeavor.  Therefore,  part 
of  the  process  of  developing  a  test  concept 
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Figure  5-3.  Test  Program  Documentation 


or  test  design  (the  distinction  between  these 
vary  from  organization  to  organization) 
should  be  to  consider  whether  the  test  will 
provide  the  information  required  by  the 
decision-makers.  In  other  words,  "Are  we 
testing  the  right  things  in  the  right  way. .  .and 
are  our  evaluations  meaningful?" 

The  test  design  is  statistical  and  analytical 
in  nature  and  should  perform  the  follow¬ 
ing  functions: 

(1 )  Structure  and  organize  the  approach 
to  testing  in  terms  of  specific  test  objectives; 

(2)  Identify  key  measures  of  effective¬ 
ness  (MOEs)  and  measures  of  performance 
(MOPs); 

(3)  Identify  the  required  data  and  dem¬ 
onstrate  how  the  data  will  be  gathered, 
stored,  analyzed  and  used  to  evaluate 
MOEs; 

(4)  Indicate  what  part  modeling  and 
simulation  will  play  in  meeting  test  objec¬ 
tives; 

(5)  Identify  the  number  and  type  of  test 
events  and  required  resources. 

The  test  design  may  serve  as  a  foundation 
for  the  more-detailed  test  plan  and  speci¬ 
fies  the  test  objectives,  events,  instrumen¬ 
tation,  methodology,  data  requirements, 
data  management  needs  and  analysis  re¬ 
quirements. 

5.5.4  Test  Plan 

The  test  plan  is  the  vehicle  that  translates  a 
test  concept  and  statistical/ analytical  test 
design  into  concrete  resources,  procedures 
and  responsibilities.  The  size  and  complex¬ 
ity  of  a  test  program  and  its  associated  test 
plan  are  determined  by  the  nature  of  the 
system  being  tested  and  the  type  of  testing 


that  is  to  be  accomplished.  Some  major 
weapons  systems  may  require  large  num¬ 
bers  of  separate  tests  to  satisfy  test  objec¬ 
tives  and,  thus,  require  a  multivolume  test 
plan;  other  testing  may  be  well-defined  by 
a  relatively  brief  test  plan.  The  test  plan  also 
provides  a  description  of  the  equipment 
configuration  and  known  limitations  to  the 
scope  of  testing.  The  type  of  information 
typically  included  in  a  test  plan  is  shown  in 
Table  5-1. 

5.5.5  Outline  Test  Plan/Test 
Program  Outline 

The  Army's  Outline  Test  Plan  (OTP)  and 
Air  Force's  Test  Program  Outline  (TPO)  are 
essential  test  planning  documents.  They 
are  formal  resource  documents  specifying 
the  resources  required  to  support  the  test. 
Since  the  OTP  or  TPO  provides  the  basis  for 
fiscal  programming  and  coordinating  the 
necessary  resources,  it  is  important  that 
these  documents  be  developed  in  advance 
and  kept  current  to  reflect  maturing  re¬ 
source  requirements  as  the  test  program 
develops.  The  Navy  makes  extensive  use  of 
the  TEMP  to  document  T&E  resource  re¬ 
quirements.  Each  Service  has  periodic  meet¬ 
ings  designed  to  review  resource  require¬ 
ments  and  resolve  problems  with  test  sup¬ 
port. 

5.5.6  Test  Reports 

5.5.6.1  Quick-Look  Reports 

Quick-look  analyses  are  expeditious  analy¬ 
ses  performed  during  testing  using  limited 
amounts  of  the  data  base.  Such  analyses 
often  are  used  to  assist  in  managing  test 
operations.  Quick-look  reports  are  used 
occasionally  to  inform  higher  authorities  of 
test  results.  Quick-look  reports  may  have 
associated  briefings  that  present  T&E  re¬ 
sults  and  substantiate  conclusions  or  rec¬ 
ommendations.  Quick- look  reports  maybe 
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Table  5-1.  Sample  Test  Plan  Contents 


PRELIMINARY  PAGES 

i.  Title  page 

ii.  Abstract 

ill.  Table  of  Contents 
iv.  Terms  and  Abbreviations 
V.  Related  Documents* 

*The  actual  number  of  these  pages  will  be  determined  by  the  length  of 
preliminary  elements  (e.g..  Table  of  Contents,  Terms  and  Abbreviations, 


preliminary  elements  (e.g..  Table  of  Contents,  Terms 
etc.). 

MAIN  BODY 

1 .  Introduction 

2.  Test  Purpose  and  Ot^ectives 

3.  Concept  of  Test  Operations 

4.  Method  of  Accomplishment 

5.  Test  Schedule 

6.  Test  Management  and  Organization 

7.  Responsibilities/Support 

8.  Personnel 

9.  Required  Test  Reports 

10.  Safety 

11.  Security 

12.  Information 

13.  Environmental  Protection 

ANNEXES 


A.  Test  Design 

B.  Data  Requirements 

C.  Instrumentation  Plan 

D.  Logistics  Support  Requirements 

E.  Reliability  and  Maintainability  Data  Plan 

F.  Intelligence/Threat  Information 
G-Z.  As  Required 

1 , 2, 3.  etc..  Detailed  Test  Procedures  (Name  of  Test) 
Distribution: 


Source:  'Standard  Procedures  for  USAF  OT&E,'  July  1974. 


generated  by  the  contractor  or  government 
agency.  They  are  of  particularly  critical 
interest  for  high-visibility  systems  that  may 
be  experiencing  some  development  diffi¬ 
culties.  Techniques  and  formats  should  be 
determined  before  the  start  of  testing.  They 
may  be  exercised  during  pretest  trials. 

5.5.6.2  Final  Test  Report 

The  final  test  report  disseminates  the  test 
information  to  decision  authorities,  pro¬ 
gram  office  staff  and  the  acquisition  com¬ 
munity.  It  provides  a  permanent  record  of 
the  execution  of  the  test  and  its  results.  The 
final  test  report  should  relate  the  test  re¬ 
sults  to  the  critical  issues  and  address  the 
objectives  stated  in  the  test  design  and  test 
plan.  A  final  test  report  may  be  separated 
into  two  sections  —  a  main  section  provid¬ 
ing  the  essential  information  about  test 
methods  and  results,  and  a  second  section 
consisting  of  supporting  appendices  to  pro¬ 
vide  details  and  supplemental  information. 
Generally,  the  following  topics  are  included 
in  the  main  body  of  the  report: 

(1)  Test  purpose 

(2)  Issues  and  objectives 

(3)  Method  of  accomplishment 

(4)  Results  (keyed  to  the  objectives  and 
issues) 

(5)  Discussion,  conclusions  and  recom¬ 
mendations. 

Appendices  of  the  final  test  report  may 
address  the  following  topics: 

(1)  Detailed  test  description 

(2)  Test  environment 

(3)  Test  organization  and  operation 


(4)  Instrumentation 

(5)  Data  collection  and  management 

(6)  Test  data 

(7)  Data  analysis 

(8)  Modeling  and  simulation 

(9)  Reliability,  availability  and  main¬ 
tainability  information 

(10)  Personnel 

(11)  Training 

(12)  Safety 

(13)  Security 

(14)  Funding 

(15)  Asset  Disposition. 

The  final  test  report  may  contain  an  evalu¬ 
ation  and  analysis  of  the  results,  or  the 
evaluation  may  be  issued  separately.  The 
analysis  tells  what  the  results  are,  whereas 
an  evaluation  tells  what  the  results  mean. 
The  evaluation  builds  on  the  analysis  and 
generalizes  from  it,  showing  how  the  re¬ 
sults  apply  outside  the  test  arena.  It  shows 
what  the  implications  of  the  test  are  and 
may  provide  recommendations.  The  evalu¬ 
ation  may  make  use  of  independent  analy¬ 
ses  of  all  or  part  of  the  data;  it  may  employ 
data  from  other  sources  and  may  use  mod¬ 
eling  and  simulation  to  generalize  the  re¬ 
sults  and  extrapolate  to  other  conditions.  In 
the  case  of  the  Army,  a  separate  Indepen¬ 
dent  Evaluation  Report  is  prepared  by  tech¬ 
nical  independent  evaluators  and  opera¬ 
tional  independent  evaluators. 
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5.6  OTHER  TEST-RELATED 
STATUS  REPORTS 

5.6.1  End  of  Test  Phase  Report 

The  Services  are  required  by  DODI  5000.2 
to  submit  to  OSD  copies  of  their  formal 
DT&E  and/or  OT&E  reports  that  are  pre¬ 
pared  at  the  end  of  each  phase  of  DT&E  or 
OT&E.  For  major  defense  acquisition  pro¬ 
grams/reviews,  draft  reports  must  be  re¬ 
ceived  by  the  Defense  Acquisition  Board 
(DAB)  executive  secretary  no  later  than  45 
days  before  a  scheduled  review. 

5.6.2  Low-Rate  Initial 
Production  Report 

Before  an  ACAT I  or  designated  program 
can  proceed  beyond  (MS  HI)  Low-Rate  Ini¬ 
tial  Production  (LRIP),  the  DOT&E  must 
submit  a  report  to  the  Secretary  of  Defense 


and  the  Senate  and  House  of  Representa¬ 
tives  Committees  on  Armed  Services  and 
on  Appropriations.  This  report  addresses 
whether  the  OT&E  performed  was  ad¬ 
equate  and  whether  the  OT&E  results  con¬ 
firm  that  the  items  or  components  tested 
are  effective  and  suitable  for  use  in  combat 
by  typical  military  users. 

5.7  SUMMARY 

A  wide  range  of  documentation  is  avail¬ 
able  to  the  test  manager  and  should  be  used 
to  develop  T&E  programs  that  address  all 
relevant  issues.  The  program  manager  must 
work  to  ensure  that  T&E  requirements  are 
considered  at  the  outset  when  the  acquisi¬ 
tion  strategy  is  formulated.  He  must  also 
require  early,  close  coordination  and  a  con¬ 
tinuing  dialogue  among  those  responsible 
for  the  SEMP,  the  ILSP  and  the  TEMP. 
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6 

TYPES  OF  TEST  AND  EVALUATION 


6.1  INTRODUCTION 

This  chapter  provides  a  brief  introduction 
to  development  test  and  evaluation  (DT &E) 
and  operational  test  and  evaluation  (OT&E) 
—  two  principal  types  of  test  and  evalua¬ 
tion  (T&E);  it  also  discusses  the  role  of 
qualification  testing  as  a  subelement  of 
development  testing.  Other  important  types 
of  T&E  are  introduced.  They  include:  multi- 
Service  testing;  joint  T&E;  live  fire  testing; 
nuclear,  chemical  and  biological  testing; 
and  nuclear  hardening  and  survivability 
testing.  As  Figure  6-1  illustrates,  DT&E  and 
OT&E  are  performed  throughout  the  ac¬ 
quisition  process  and  identified  by  nomen¬ 
clature  that  may  change  with  the  phase  of 
the  acquisition  cycle  in  which  they  occur. 

6.2  DEVELOPMENT  TEST 
AND  EVALUATION  (DT&E) 

Development  test  and  evaluation  is  T&E 
conducted  throughout  the  acquisition  pro¬ 
cess  to  assist  in  engineering  design  and 
development  and  to  verify  that  technical 
performance  specifications  have  been  met. 
The  DT&E  is  planned  and  monitored  by 
the  developing  agency  and  is  normally  con¬ 
ducted  by  the  contractor.  However,  the 
development  agency  may  perform  techni¬ 
cal  compliance  tests  before  OT&E.  It  in¬ 
cludes  the  T &E  of  components,  subsystems, 
preplanned  product  improvement  (P^I) 
changes,  hardware /software  integration 
and  production  qualification  testing.  It 
encompasses  the  use  of  models,  simula¬ 
tions  and  test-beds,  and  prototypes  or  full- 


scale  engineering  development  models  of 
the  system.  Development  test  and  evalua¬ 
tion  may  involve  a  wide  degree  of  test 
complexity,  depending  upon  the  type  of 
system  or  test  article  under  development; 
e.g.,  tests  of  electronic  breadboards  or 
brassboards,  components,  subsystems  or 
experimental  prototypes. 

Development  test  and  evaluation  supports 
the  system  design  process  through  a  test- 
analyze-fix-retest  approach  that  involves 
both  contractor  and  government  person¬ 
nel.  Because  contractor  testing  plays  a  piv¬ 
otal  role  in  the  total  test  program,  it  is 
important  the  contractor  establishes  an  in¬ 
tegrated  test  plan  early  to  ensure  that  the 
scope  of  the  contractor's  test  program  satis¬ 
fies  government  and  contractor  test  objec¬ 
tives. 

The  program  manager  remains  responsible 
for  the  ultimate  success  of  the  overall  pro¬ 
gram.  He  and  the  test  specialists  on  his  staff 
must  foster  an  environment  that  provides 
the  contractor  with  sufficient  latitude  to 
pursue  innovative  solutions  to  technical 
problems  and,  at  the  same  time,  provides 
the  data  needed  to  make  rational  trade-off 
decisions  between  cost,  schedule  and  per¬ 
formance  as  the  program  progresses. 

6.2.1  Production  Qualification 
Tests  (PQT) 

Qualification  testing  is  a  form  of  develop¬ 
ment  testing  that  verifies  the  design  and 
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manufacturing  process.  Production  quali¬ 
fication  tests  are  formal  contractual  tests 
that  confirm  the  integrity  of  the  system 
design  over  the  specified  operational  and 
envirorunental  range.  These  tests  usually 
use  prototype  or  preproduction  hardware 
fabricated  to  the  proposed  production  de¬ 
sign  specifications  and  drawings.  Such  tests 
include  contractual  reliability  and  main¬ 
tainability  demonstration  tests  required 
before  production  release.  Production  quali¬ 
fication  T&E  must  be  completed  before 
Milestone  III. 

Production  qualification  tests  may  be  con¬ 
ducted  on  low-rate  initial  production  items 
to  ensure  the  effectiveness  of  the  manufac¬ 
turing  process,  equipment  and  procedures. 
These  tests  are  conducted  on  each  item  or  a 
sample  lot  taken  at  random  from  the  first 
production  lot  and  are  repeated  if  the  pro¬ 
cess  or  design  is  changed  significantly  or  a 
second  or  alternative  source  is  brought  on 
line.  These  tests  are  also  conducted  against 
contractual  design  and  performance  re¬ 
quirements. 

6.3  OPERATIONAL  TEST 
AND  EVALUATION  (OT&E) 

6.3.1  The  Difference  Between 
Development  and  Operational  Testing 

Air  Force  Manual  55-43,  published  in  June 
1979,  once  contained  the  following  account 
of  the  first  OT &E;  this  anecdote  serves  as  an 
excellent  illustration  of  the  difference  be¬ 
tween  developmmt  and  operational  test¬ 
ing: 

The  test  and  evaluation  of  aircraft  and 
air  weapon  systems  started  with  the 
contract  awarded  to  the  Wright  broth¬ 
ers  in  1908.  This  contract  specified  a 
craft  which  would  lift  two  men  with  a 
total  weight  of  350  pounds,  carry 
enough  fuel  for  a  flight  of  125  miles. 


and  fly  40  miles  per  hour  in  still  air. 
The  contract  also  required  that  testing 
be  conducted  to  assure  this  capability. 

What  we  now  call  development  test 
and  evaluation  (DT&E)  was  satisfied 
when  the  Wright  brothers  (the  devel¬ 
oper)  demonstrated  that  their  airplane 
could  meet  those  first  contract  specifi¬ 
cations.  However,  no  immediate  mili¬ 
tary  mission  had  been  conceived  for 
the  Wright  Flyer.  It  was  shipped  to 
Fort  Sam  Houston,  Texas,  where  Cap¬ 
tain  Benjamin  D.  Foulois,  the  pilot, 
had  orders  to  “teach  himself  to  fly." 
He  had  to  determine  the  airplane's 
performance,  how  to  maintain  it,  and 
the  kind  of  organization  that  would 
use  it.  Cavalry  wagon  masters  had  to 
be  trained  as  airplane  mechanics,  and 
Captain  Foulois  was  his  own  instruc¬ 
tor  pilot. 

In  the  process.  Captain  Foulois  sub¬ 
jected  the  Wright  Flyer  to  test  and 
evaluation  under  operational  condi¬ 
tions.  Foulois  soon  discovered  opera¬ 
tional  deficiencies.  For  example,  there 
was  no  seat  on  the  airplane.  During 
hard  landings,  Foulois'  130  pound 
frame  usually  parted  company  from 
the  airplane.  To  correct  the  problem, 
Foulois  bolted  an  iron  tractor  seat  to 
the  airplane.  The  seat  helped,  but 
Foulois  still  toppled  from  his  perch  on 
occasion.  As  a  further  improvement, 
Foulois  looped  his  Sam  Browne  belt 
through  the  seat  and  strapped  himself 
in.  Ever  since  then,  contoured  seats 
and  safety  belts  —  a  product  of  this 
earliest  "operational"  test  and  evalua¬ 
tion  -have  been  part  of  the  military 
airplane. 

Captain  Foulois'  experience  may  seem  hu¬ 
morous  now,  but  it  dramatically  illustrates 
the  need  for  operational  testing.  It  also 
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shows  that  operational  testing  has  been 
going  on  for  a  long  time. 

As  shown  in  Table  6-1  where  development 
testing  is  focused  on  meeting  detailed  tech¬ 
nical  specifications,  the  operational  test  fo¬ 
cuses  on  the  actual  fimctioning  of  the  equip¬ 
ment  in  a  realistic  combat  environment  in 
which  the  equipment  must  interact  with 
humans  and  peripheral  equipment.  Where 
DT&E  and  OT&E  are  separate  activities 
and  are  conducted  by  different  test  com¬ 
munities,  the  communities  must  interact 
frequently  and  are  generally  complemen¬ 
tary.  The  DT&E  provides  a  view  of  the 
potential  to  reach  technical  objectives,  and 
OT&E  provides  an  assessment  of  the 
system's  potential  to  satisfy  user  require¬ 
ments. 

6.3.2  The  Purpose  of  Operational 
Test  and  Evaluation 

Operational  Test  and  Evaluation  is  defined 
in  Title  10,  U.S.  code: 

Definitions  of  operational  effectiveness  and 
operational  suitability,  outlined  in  DoDI 
5000.2,  PT  15  are  listed  below: 

Operational  Effectiveness:  The  overall  de¬ 
gree  of  mission  accomplishment  of  a  sys¬ 
tem  when  used  by  representative  person¬ 
nel  in  the  environment  planned  or  expected 
(e.g.  natural,  electronic,  threat  etc.)  for  op¬ 
erational  employment  of  the  system  (.on- 
sidering  organization,  doctrine,  tactics,  sur¬ 
vivability,  vulnerability,  and  threat  (in¬ 
cluding  countermeasures,  initial  nuclear 
weapons  effects,  nuclear,  biological  and 
chemical  contamination  (NBCC)  threats). 

Operational  Suitability:  The  degree  to  which 
a  system  can  be  placed  satisfactorily  in  field 
use  with  consideration  given  to  availabil¬ 
ity,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage 


rates,  maintainability,  safety,  human  fac¬ 
tors,  manpower  supportability,  logistics 
supportability,  natural  environmental  ef¬ 
fects  and  impacts,  documentation  and  train¬ 
ing  requirements. 

In  each  of  the  Services,  operational  testing 
is  conducted  under  the  auspices  of  an  orga¬ 
nization  that  is  independent  of  the  devel¬ 
opment  agency,  in  as  operationally  realis¬ 
tic  environments  as  possible,  with  hostile 
forces  representative  of  the  anticipated 
threat  and  with  typical  users  operating  and 
maintaining  the  system.  In  other  words, 
"OT&E  is  conducted  to  ensure  that  new 
systems  meet  the  user's  requirements,  op¬ 
erate  satisfactorily,  and  are  supportable 
under  actual  field  conditions,"  (Reference 
2).  The  major  questions  addressed  in  OT&E 
are  shown  in  Figure  6-2. 

Early  Operational  Assessment,  Operational 
Assessment  (EOA,  OA):  The  operational 
test  normally  takes  place  during  the  con¬ 
cept  exploration  /  definition  and  demonstra¬ 
tion/validation  phases  and  is  used  to  pro¬ 
vide  an  early  assessment  of  potential  op¬ 
erational  effectiveness  and  suitability  and 
to  project  the  system's  potential  to  meet  the 
user's  requirements. 

6.3.3  Initial  Operational 
Test  and  Evaluation 

The  OT&E  performed  in  support  of  the 
full-rate  production  decision  (Milestone 
(MS)  III)  is  known  as  Initial  Operational 
Test  and  Evaluation  (lOT&E).  The  lOT&E 
begins  during  the  Engineering  and  Manu¬ 
facturing  Development  (EMD)  Phase  and 
ends  before  the  full-rate  production  deci¬ 
sion.  More  than  one  lOT&E  may  be  con¬ 
ducted  on  the  system.  The  operational  test 
is  conducted  on  a  production  representa¬ 
tive  system  using  typical  operational  per¬ 
sonnel  in  as  realistic  a  scen.''rio  as  possible 
to  verify  a  system's  operatior.al  effective- 
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igure  6-2.  Sample  Hierarchy  of  Questions  Addressed  In  Operational  Test  and  Evaluation 


ness  and  suitability  and  to  ensure  that  the 
system  meets  operational  thresholds. 

6.3.4  Follow-on  Operational 
Test  and  Evaluation 

The  OT&E  performed  after  the  start  of  full- 
rate  production  may  be  called  follow-on 
operational  test  and  evaluation  (FOT&E) 
and  is  conducted  during  production  and 
deployment.  It,  too,  is  sometimes  divided 
into  two  separate  activities.  Preliminary 
FOT&E  is  normally  conducted  after  the 
initial  operational  capability  is  attained  to 
assess  full  system  capability.  It  is  conducted 
by  the  OT&E  organization  to  verify  the 
correction  of  deficiencies,  if  required,  and 
to  assess  system  training  and  logistics  sta¬ 
tus.  Subsequent  FOT&E  is  conducted  on 
production  items  throughout  the  life  of  a 
system.  The  results  are  used  to  refine  esti¬ 
mates  of  operational  effectiveness  and  suit¬ 
ability;  to  update  training,  tactics,  tech¬ 
niques  and  doctrine;  and  to  identify  opera¬ 
tional  deficiencies  and  evaluate  modifica¬ 
tions.  This  FOT&E  is  conducted  using  the 
operating  command. 

6.4  MULTI-SERVICE 
TEST  AND  EVALUATION 

Multi-Service  test  and  evaluation  is  T&E 
conducted  on  a  system  being  acquired  for 
use  by  more  than  one  Service.  All  affected 
Services  and  their  respective  operational 
test  agencies  participate  in  planning,  con¬ 
ducting,  reporting  and  evaluating  the  multi- 
Service  test  program.  One  Service  is  desig¬ 
nated  the  lead  Service  and  is  responsible 
for  the  management  of  the  program.  The 
lead  Service  is  charged  (by  DoDI  5000.2) 
with  the  preparation  and  coordination  of  a 
single  report  that  reflects  the  system's  op¬ 
erational  effectiveness  and  suitability  for 
each  Service. 

The  management  challenge  in  a  multi-Ser- 
vice  test  program  stems  from  the  fact  that 


the  items  undergoing  testing  will  not  nec¬ 
essarily  be  used  by  each  of  the  Services  for 
identical  purposes.  Differences  among  the 
Services  usually  exist  in  performance  crite¬ 
ria,  tactics,  doctrine,  configuration  of  arma¬ 
ment  or  electronics  and  the  operating  envi¬ 
ronment.  As  a  result,  a  deficiency  or  dis¬ 
crepancy  considered  disqualifying  by  one 
Service  is  not  necessarily  disqualifying  for 
all  Services.  It  is  incumbent  upon  the  lead 
Service  to  establish  a  discrepancy  report¬ 
ing  system  that  permits  each  participating 
Service  to  document  all  discrepancies  noted. 
At  the  conclusion  of  a  multi-Service  T&E, 
each  participating  OT&E  agency  prepares 
an  independent  evaluation  report  in  its 
own  format  and  submits  that  report  through 
its  normal  Service  channels.  The  lead  Ser¬ 
vice  OT&E  agency  prepares  the  documen¬ 
tation  that  goes  forward  to  the  Defense 
Acquisition  Board;  this  documentation  is 
coordinated  with  all  participating  OT&E 
agencies. 

6.5  JOINT  TEST 
AND  EVALUATION 

Joint  T&E  is  not  the  same  as  multi-Service 
T&E.  Joint  T&E  is  a  specific  program  activ¬ 
ity  sponsored  and  funded  by  the  Office  of 
the  Secretary  of  Defense  (OSD).  Joint  T&E 
programs  are  not  acquisition  oriented;  they 
are  a  means  of  examining  joint-Service  tac¬ 
tics  and  doctrine.  Past  joint-test  programs 
have  been  conducted  to  provide  informa¬ 
tion  required  by  the  Congress,  the  OSD,  the 
commanders  of  the  Unified  and  Specified 
Commands  and  the  Services.  Joint  tests  are 
usually  characterized  as  either  Joint  Devel¬ 
opment  T&E  or  Joint  Operational  T&E.  Joint 
development  T&Es  focus  on  obtaining  in¬ 
formation  on  system  requirements,  system 
performance,  system  interoperability,  tech¬ 
nical  concepts,  technical  improvements,  im¬ 
proved  testing  methodologies  or  test  re¬ 
source  requirements. 
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Joint  operational  tests  and  evaluations  are 
conducted  using  actual  fielded  equipment, 
simulators  or  surrogate  equipment  in  an 
exercise  or  operational  environment  to  ob¬ 
tain  data  pertinent  to  operational  doctrine, 
tactics  and  procedures. 

The  OSD  reviews  candidate  nominations 
for  joint  test  programs  each  year;  and,  if  a 
proposal  is  deemed  appropriate  by  the  fea¬ 
sibility  study,  a  lead  ^rvice  is  selected  and 
tasked  to  plan  and  execute  the  program 
using  a  test  force  of  participating  Service 
personnel. 

The  commanders  of  the  four-Service  op¬ 
erational  test  agencies  —  the  Army  Opera¬ 
tional  Test  and  Evaluation  Command 
(OPTEC),  the  Navy  Operational  Test  and 
Evaluation  Force  (OI^EVFOR),  the  Air 
Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  and  the  Marine  Corps 
Operational  Test  and  Evaluation  Activity 
(MCOTEA)  —  have  signed  a  Memoran¬ 
dum  of  Agreement  on  Multi-Service  OT&E 
and  Joint  T&E  (Reference  37)  that  stipu¬ 
lates  how  both  types  of  programs  are  to  be 
managed. 

6.6  LIVE  FIRE  TESTING 

The  Live  Fire  Test  Program  was  mandated 
by  the  Congress  in  the  National  Defense 
Authorization  Act  for  Fiscal  1987  (Public 
Law  99-661)  passed  in  November  1986. 
Specifically,  this  law  stipulated  that  a  ma¬ 
jor  defense  acquisition  program  may  not 
proceed  beyond  low-rate  initial  produc¬ 
tion  until  realistic  survivability  or  (in  the 
case  of  missiles  and  munitions)  lethality 
testing  has  been  completed. 

In  1984,  before  the  passage  of  this  legisla¬ 
tion,  the  OSD  had  chartered  a  joint  test 
program  designed  to  address  similar  ques¬ 
tions  relative  to  systems  already  in  field 
use.  This  program,  the  Joint  Live  Fire  Test, 


was  initially  divided  into  two  distinct  parts: 
Armor/Antiarmor  and  Aircraft.  The 
program's  objectives  are  to: 

•  Gather  empirical  data  on  the  vulner¬ 
ability  of  existing  U.S.  systems  to  Soviet 
weapons; 

•  Gather  empirical  data  on  the  lethality 
of  existing  U.S.  weapons  against  Soviet 
systems; 

•  Provide  insights  into  the  design  changes 
necessary  to  reduce  vulnerabilities  and 
improve  lethalities  of  existing  U.S.  weapon 
systems; 

•  Calibrate  current  vulnerability  and 
lethality  models. 

The  legislated  Live  Fire  Test  (LFT)  Program 
complements  the  older  Joint  Live  Fire  (JLF) 
Program.  While  the  JLF  Program  was  de¬ 
signed  to  test  systems  that  were  fielded 
before  being  completely  tested,  the  spirit 
and  intent  of  the  LFT  legislation  is  to  avoid 
the  need  to  play  "catch-up."  This  program 
not  only  requires  the  Services  to  test  their 
weapons  systems  as  early  as  possible 
against  the  expected  combat  threat  but  also 
before  MS  III  to  identify  design  characteris¬ 
tics  that  cause  undue  combat  damage  or 
measure  munitions  lethality.  Remedies  for 
deficiencies  can  entail  required  retrofits, 
production  stoppages  or  other  more  time- 
consuming  solutions.  The  essential  feature 
of  live  fire  testing  is  that  appropriate  threat 
munitions  are  fired  against  a  major  U.S. 
system  configured  for  combat  to  test  its 
vulnerability  and/or  that  a  major  U.S.  mu¬ 
nitions  or  missile  is  fired  against  a  threat 
target  configured  for  combat  to  test  the 
lethality  of  the  munitions  or  missile. 

Live  Fire  Test  and  Evaluation  Guidelines 
were  first  issued  by  the  Deputy  Director, 
T&E  (Live  Fire  Testing)  in  May  1987  to 
supplement  DOD  Test  and  Evaluation 
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Master  Plan  guidelines  (DOD  5000.2-M)  in 
areas  pertaining  to  live  fire  testing  (Refer¬ 
ence  34).  These  guidelines  encompass  all 
major  defense  acquisition  programs  and 
define  LFT  requirements. 

6.7  NUCLEAR,  BIOLOGICAL  AND 
CHEMICAL  WEAPONS  TESTING 

The  testing  of  nuclear,  biological  and  chemi¬ 
cal  (NBC)  weapons  is  highly  specialized 
and  regulated.  Program  managers  involved 
in  these  areas  are  advised  to  consult  au¬ 
thorities  within  their  chain  of  command  for 
the  specific  directives,  instructions  and 
regulations  that  apply  to  their  individual 
situations.  Nuclear  weapons  tests  are  di¬ 
vided  into  categories  in  which  the  respon¬ 
sibilities  of  the  Department  of  Energy 
(DOE),  the  Defense  Nuclear  Agency  (DNA) 
and  the  military  Services  are  clearly  as¬ 
signed.  The  DOE  is  responsible  for  nuclear 
warhead  technical  tests;  the  DNA  is  re¬ 
sponsible  for  nuclear  weapons  effects  tests. 
The  Services  are  responsible  for  the  testing 
of  Service-developed  components  of 
nuclear  subsystems.  All  nuclear  tests  are 
conducted  within  the  provisions  of  the  Lim¬ 
ited  Test  Ban  Treaty  that  generally  restricts 
nuclear  detonations  to  the  underground 
environment.  Nuclear  weapons  testing  re¬ 
quires  extensive  coordination  between  Ser¬ 
vice  and  DOE  test  persoimel  (Reference 
55). 

Since  the  United  States  signed  and  ratified 
the  Geneva  Protocol  of 1925,  U.S.  policy  has 
been  never  to  be  the  first  to  use  lethal  chemi¬ 
cal  weapons;  it  may,  however,  retaliate  with 
chemical  weapons  if  so  attacked.  With  the 
signing  and  ratification  of  the  1972  Biologi¬ 
cal  and  Toxin  Weapon  Convention,  the 
United  States  formally  adopted  the  posi¬ 
tion  that  it  would  not  employ  biological  or 
toxin  weapons  under  any  circumstances. 
All  such  weapons  were  reported  destroyed 
in  the  early  '70s  (Reference  38). 


Regarding  retaliatory  capability  against 
chemical  weapons,  the  Servirs  Secretaries 
are  responsible  for  ensuring  that  their  or¬ 
ganizations  establish  requirements  and 
determine  the  military  characteristics  of 
chemical  deterrent  items  and  chemical  de¬ 
tense  items.  Tlie  Army  has  been  designated 
the  DOD  executive  agent  for  DOD  chemi¬ 
cal  warfare,  research,  development  and 
acquisition  programs  (Reference  39). 

United  States  policy  on  chemical  warfare 
seeks  to: 

•  Deter  the  use  of  chemical  warfare  weap¬ 
ons  by  other  nations; 

•  Provide  the  capability  to  retaliate  if 
deterrence  fails; 

•  Achieve  the  early  termination  of  chemi¬ 
cal  warfare  at  the  lowest  possible  intensity 
(Reference  39). 

In  addition  to  the  customary  development 
tests  (conducted  to  determine  if  a  weapon 
meets  technical  specifications)  and  opera¬ 
tional  tests  (conducted  to  determine  if  a 
weapon  will  be  useful  in  combat),  chemical 
weapons  testing  involves  two  types  of 
chemical  tests  —  chemical  mixing  and 
biotoxicity.  Chemical-mixing  tests  are  con¬ 
ducted  to  obtain  information  on  the  binary 
chemical  reaction.  Biotoxicity  tests  are  per¬ 
formed  to  assess  the  potency  of  the  agent 
generated.  Chemical  weapons  testing,  of 
necessity,  relies  heavily  on  the  use  of  non¬ 
toxic  stimulants,  since  such  substances  are 
more  economical  and  less  hazardous  and 
open-air  testing  of  live  agents  has  been 
restricted  since  1969  (Reference  39). 

6.8  NUCLEAR  HARDNESS 
AND  SURVIVABILITY  TESTING 

Nuclear  hardness  is  a  quantitative  descrip¬ 
tion  of  the  physical  attributes  of  a  system  or 
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component  that  will  allow  it  to  survive  in  a 
given  nuclear  environment.  Nuclear  sur¬ 
vivability  is  the  capability  of  a  system  to 
survive  in  a  nuclear  environment  and  to 
accomplish  a  mission.  Department  of  De¬ 
fense  policy  requires  the  incorporation  of 
nuclear  hardness  and  survivability  features 
in  the  design,  acquisition  and  operation  of 
major  and  nonmajor  systems  that  must 
perform  critical  missions  in  nuclear  con¬ 
flicts.  Nuclear  hardness  levels  must  be  quan¬ 
tified  and  validated  (Reference  12). 

The  T&E  techniques  used  to  assess  nuclear 
hardness  and  survivability  include:  nuclear 
testing,  physical  testing  in  a  simulated  en¬ 
vironment,  modeling,  simulation  and 
analysis.  Although  nuclear  tests  provide  a 
high  degree  of  fidelity  and  valid  results  for 
survivability  evaluation,  they  are  not  prac¬ 
tical  for  most  systems  due  to  cost,  long  lead 
times  and  international  treaty  constraints. 
Underground  testing  is  available  only  on  a 
prioritized  basis  for  critical  equipment  and 
components  and  is  subject  to  a  frequently 
changing  test  schedule.  Physical  testing 


provides  an  opportunity  to  observe  per¬ 
sonnel  and  equipment  in  a  simulated 
nuclear  environment.  Modeling,  simula¬ 
tion  and  analysis  are  particularly  useful  in 
the  early  stages  of  development  to  provide 
early  projections  before  system  hardware 
is  available.  These  methods  are  also  used  to 
furnish  assessments  in  an  area  that,  be¬ 
cause  of  safety  or  testing  limitations,  can¬ 
not  be  directly  observed  through  nuclear  or 
physical  testing. 

6.9  SUMMARY 

Test  and  evaluation  is  a  technique  used  to 
address  critical  performance  questions 
during  system  development.  These  ques¬ 
tions  may  involve  several  issues  including: 
technical  (development  testing);  effective¬ 
ness,  suitability  and  supportability  (opera¬ 
tional  testing);  those  affecting  more  than 
one  Service  (multi-Service  and  joint  test¬ 
ing);  vulnerability  and  lethality  (live  fire 
testing),  nuclear  survivability;  or  the  use  of 
other  than  conventional  weapons  (i.e., 
nuclear,  biological  or  chemical). 
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MODULE 

Developmental 
Test  and  Evaluation 


Material  acquisition  is  an  iterative  process 
of  designing,  building,  testing,  identifying 
deficiencies,  fixing,  retesting  and  repeat¬ 
ing.  Development  Test  and  Evaluation 
(DT&E)  is  an  important  aspect  of  this  pro¬ 
cess.  The  DT&E  is  performed  in  the  fac¬ 
tory,  laboratory  and  on  the  proving  ground. 
It  is  conducted  by  subcontractors,  as  they 
are  developing  the  components  and  sub- 
assembly;  the  prime  contractor,  as  he  as¬ 
sembles  the  components  and  ensures  inte¬ 
gration  of  the  system;  and  by  the  govern¬ 
ment,  to  demonstrate  how  well  the  weapon 
system  meets  its  technical  and  operational 
requirements.  This  module  describes  de¬ 
velopment  testing  and  the  various  types  of 
activities  it  involves.  The  module  also  dis¬ 
cusses  how  development  testing  is  used  to 
support  the  technical  review  process. 


7 

INTRODUCTION  TO  DEVELOPMENT 
TEST  AND  EVALUATION 


7.1  INTRODUCTION 

Development  test  and  evaluation  (DT&E) 
is  the  test  and  evaluation  conducted  to 
demonstrate  that  the  engineering  design 
and  development  process  is  complete.  It  is 
used  by  the  contractor  to  reduce  risk,  vali¬ 
date  and  qualify  the  design,  and  ensure 
that  the  product  is  ready  for  government 
acceptance.  The  DT&E  results  are  evalu¬ 
ated  to  ensure  that  design  risks  have  been 
minimized  and  the  system  will  meet  speci¬ 
fications.  The  results  are  also  used  to  esti¬ 
mate  the  system's  military  utility  when  it  is 
introduced  into  service.  Development  test 
and  evaluation  serves  a  critical  purpose  in 
reducing  the  risks  of  development  by  test¬ 
ing  selected  high-risk  components  or  sub¬ 
systems.  Finally,  DT&E  is  the  government 
developing  agency  tool  used  to  confirm 
that  the  system  performs  as  technically 
specified  and  that  the  system  is  ready  for 
field  testing.  This  chapter  provides  a  gen¬ 
eral  discussion  of  contractor  and  govern¬ 
ment  DT&E  activities,  stresses  the  need  for 
an  integrated  test  program,  describes  some 
special-purpose  development  tests  (DTs) 
and  discusses  several  factors  that  may  in¬ 
fluence  the  extent  and  scope  of  the  DT&E 
program. 

7.2  DT&E  RESPONSIBILITIES 

As  illustrated  in  Figure  7-1,  the  primary 
participants  in  testing  are  the  prime  con¬ 
tractor,  subcontractor.  Service  materiel  de¬ 
veloper  or  developing  agency  and  the  op¬ 


erational  test  and  evaluation  (OT&E) 
agency.  In  some  Services,  there  are  also 
independent  evaluation  organizations  that 
assist  the  testing  organization  in  designing 
and  evaluating  development  tests.  As  the 
figure  shows,  system  development  testing 
is  performed  principally  by  contractors 
during  the  early  development  stages  of  the 
acquisition  cycle  and  by  government  test/ 
evaluation  organizations  during  the  later 
phases. 

Army  testing  of  the  Advanced  Attack  He¬ 
licopter  illustrates  the  type  of  develop¬ 
ment  testing  performed  by  contractors  and 
the  relationship  of  this  type  of  testing  to 
government  DT&E  activities. 

During  the  contractor  competitive  Phase  I 
testing  of  the  Army  Advanced  Attack  He¬ 
licopter  (AAH),  prime  contractor  and 
subcontractor  testing  included  design  sup¬ 
port  tests,  testing  of  individual  components, 
establishing  limited  fatigue  lives,  and  bench 
testing  of  dynamic  components  to  demon¬ 
strate  sufficient  structural  integrity  to  con¬ 
duct  the  Army  competitive  flight  test  pro¬ 
gram.  Complete  dynamic  system  testing 
was  conducted  utilizing  ground  test  ve¬ 
hicles.  Besides  supporting  the  contractor's 
development  effort,  these  tests  provided 
information  for  the  Army  technical  review 
process  as  the  systems,  preliminary  and 
critical  design  reviews  were  conducted. 
Following  successful  completion  of  the 
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Figure  7-1 .  Contractor’s  Integrated  Test  Requirements 


ground  test  vehicle  qualification  testing, 
first  flights  were  conducted  on  the  two 
types  of  competing  helicopters.  Each  air¬ 
craft  was  being  flown  300  hours  before 
delivery  of  two  of  each  competing  aircraft 
to  the  Army.  The  contractor  flight  testing 
was  oriented  toward  flight-envelope  de¬ 
velopment,  demonstration  of  structural 
integrity,  and  evaluation  and  verification 
of  aircraft  flight  handling  qualities.  Some 
weapons  system  testing  was  conducted 
during  this  phase.  Government  testers  used 
much  of  the  contractor's  testing  data  to 
develop  the  test  data  matrices  as  part  of  the 
government's  DT  and  operational  test  (OT) 
planning  efforts.  The  use  of  contractor  test 
data  reduced  the  testing  required  by  the 
government  and  added  validity  to  the  sys¬ 
tems  already  tested  and  data  received  from 
other  sources. 

7.2.1  Contractor  Testing 

Materiel  development,  testing  and  evalua¬ 
tion  are  an  iterative  process  in  which  a 
contractor  designs  hardware  and  software, 
evaluates  performance,  makes  changes  as 
necessary,  and  retests  for  performance  and 
technical  compliance.  Contractor  testing 
plays  a  primary  role  in  the  total  test  pro¬ 
gram,  and  the  results  of  contractor  tests  are 
useful  to  the  government  evaluator  in  sup¬ 
porting  government  test  objectives.  It  is 
important  that  government  evaluators,  as 
appropriate,  oversee  contractor  system  tests 
and  use  test  data  from  them  to  address 
government  testing  issues.  It  is  not  uncom¬ 
mon  for  contractor  testing  to  be  conducted 
at  government  test  facilities,  since  contrac¬ 
tors  often  do  not  have  the  required  special¬ 
ized  facilities  (e.g.,  for  testing  hazardous 
components  or  for  missile  flight  tests).  This 
enables  government  evaluators  to  monitor 
the  tests  more  readily  and  increases  gov¬ 
ernment  confidence  in  the  test  results. 

Normally,  a  Request  For  Proposal  (RFP) 
requires  that  the  winning  contractor  sub¬ 


mit  an  Engineering  Design  Test  Plan  within 
60  to  90  days  after  contract  initiation  for 
coordination  with  government  test  agen¬ 
cies  and  approval.  This  test  plan  should 
include  testing  required  by  the  Statement 
of  Work  (SOW),  specifications,  various  MIL- 
SPECs  and  MIL-STDs,  and  testing  expected 
as  part  of  the  engineering  development 
and  integration  process.  When  approved, 
the  contractor's  test  program  automatically 
becomes  part  of  the  development  agency's 
Integrated  Test  Plan. 

If  the  contractor  has  misinterpreted  the 
RFP  requirements  and  the  Engineering 
Design  Test  Plan  does  not  satisfy  govern¬ 
ment  test  objectives,  the  iterative  process  of 
amending  the  contractor's  test  program 
begins.  This  iterative  process  must  be  ac¬ 
complished  within  limited  bounds  so  the 
contractor  can  meet  the  test  objectives  with¬ 
out  significant  effects  on  contract  cost, 
schedule  or  scope. 

7.2.2  Government  Testing 

Government  testing  is  performed  to:  dem¬ 
onstrate  how  well  the  materiel  system  meets 
its  technical  compliance  requirements,  pro¬ 
vide  data  to  assess  developmental  risk  for 
decision-making,  verify  that  the  technical 
and  support  problems  identified  in  previ¬ 
ous  testing  have  been  corrected,  and  en¬ 
sure  that  all  critical  issues  to  be  resolved  by 
testing  have  been  adequately  considered. 
All  previous  testing,  from  the  contractor's 
bench  testing  through  development  agency 
testing  of  representative  prototypes,  is 
considered  during  government  evaluation. 

Government  materiel  development  orga¬ 
nizations  include  major  materiel  acquisi¬ 
tion  commands  and,  in  some  cases,  opera¬ 
tional  commands.  The  materiel  acquisition 
commands  have  test  and  evaluation  (T&E) 
organizations  that  conduct  government 
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development  testing.  In  addition  to 
monitoring  and  participating  in  contractor 
testing,  these  organizations  conduct  devel¬ 
opment  testing  on  selected  high-concem 
areas  to  evaluate  the  adequacy  of  systems 
engineering,  design,  development  and  per¬ 
formance  to  specifications.  The  program 
management  office  (PMO)  must  be  in¬ 
volved  in  all  stages  of  testing  that  these 
organizations  perform. 

In  turn,  the  materiel  development/ test  and 
evaluation  agencies  conduct  T&E  of  the 
systems  in  the  development  stage  to  ensure 
they  meet  technical  and  operational  re¬ 
quirements.  These  organizations  operate 
government  proving  grounds,  test  facili¬ 
ties  and  labs;  and  they  must  be  responsive 
to  the  needs  of  the  program  manager  (PM) 
by  providing  test  facilities,  personnel  and 
data  acquisition  services,  as  required. 

7.2.3  Program  Manager's  Role 

The  PM  is  responsible  for  coordinating  the 
total  T&E  program.  He  performs  this  task 
with  the  assistance  of  the  T&E  working 
group  whose  members  are  assembled  from 
development  agency,  user,  technical  and 
operational  T&E,  logistics,  and  training  or¬ 
ganizations.  The  PM  must  remain  active  in 
all  aspects  of  testing  including  planning, 
funding,  resourcing,  execution  and  report¬ 
ing.  He  plays  an  important  role  as  the  inter¬ 
face  between  the  contractor  and  the  gov¬ 
ernment  testing  community.  Recent  em¬ 
phasis  on  early  T&E  highlights  a  need  for 
early  government  tester  involvement  in 
contractor  testing.  For  example,  during  de¬ 
velopment  of  the  AAH  test,  it  was  found 
that  having  program  management  person¬ 
nel  on  the  test  sites  improved  test  continu¬ 
ity,  facilitated  the  flow  of  spare  and  repair 
parts,  provided  a  method  of  monitoring 
contractor  performance,  and  kept  the  Ser¬ 
vice  headquarters  informed  with  timely 
status  reports. 


7.3  TEST  PROGRAM 
INTEGRATION 

During  the  development  of  a  weapon  sys¬ 
tem,  there  are  a  number  of  tests  conducted 
by  subcontractors,  the  prime  contractor  and 
the  government.  To  ensure  these  tests  are 
properly  time-phased,  that  adequate  re¬ 
sources  are  available,  and  to  minimize  un¬ 
necessary  testing,  a  coordinated  test  pro¬ 
gram  must  be  developed  and  followed. 
The  Test  and  Evaluation  Master  Plan 
(TEMP)  normally  does  not  provide  a  suffi¬ 
cient  level  of  detail  concerning  contractor 
or  subcontractor  tests.  A  contractor  or  PMO 
Integrated  Test  Plan  must  also  be  devel¬ 
oped  to  describe  these  tests. 

7.3.1  Integrated  Test  Plan 

The  Integrated  Test  Plan  (FTP)  is  used  to 
record  the  individual  test  plans  for  the 
subcontractor,  prime  contractor  and  gov¬ 
ernment.  The  prime  contractor  should  be 
contractually  responsible  for  the  prepara¬ 
tion  and  updating  of  the  FTP,  and  the  con¬ 
tractor  and  Service-developing  agency 
should  ensure  that  it  remains  current.  The 
FTP  includes  all  developmental  tests  that 
will  be  performed  by  the  prime  contractor 
and  the  subcontractors  at  both  the  system 
and  subsystem  levels.  It  is  a  detailed,  work¬ 
ing-level  document  that  assists  in  identify¬ 
ing  risk  as  well  as  duplicative  or  missing 
test  activities.  A  well-maintained  FFP  facili¬ 
tates  the  most  efficient  use  of  test  resources. 

7.3.2  Single  Integrated  Test  Policy 

Most  Services  have  adopted  a  single  inte¬ 
grated  contractor/ government  test  policy, 
thereby  reducing  much  of  the  government 
testing  requirements.  This  policy  stresses 
independent  government  evaluation  and 
permits  an  evaluator  to  monitor  contractor 
and  government  test  programs  and  evalu¬ 
ate  the  system  from  an  independent  per- 
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spective.  The  policy  stresses  the  use  of  all 
available  test  data  for  system  evaluation. 

7.4  AREAS  OF  DT&E  FOCUS 

7.4.1  Life  Testing 

Life  testing  is  performed  to  assess  the  ef¬ 
fects  of  long-term  exposure  to  various  por¬ 
tions  of  the  anticipated  environment.  These 
tests  are  used  to  ensure  the  system  will  not 
fail  prematurely  due  to  metal  fatigue,  com¬ 
ponent  aging  or  other  problems  caused  by 
long-term  exposure  to  environmental  ef¬ 
fects.  It  is  important  that  the  requirements 
for  life  testing  are  identified  early  and  inte¬ 
grated  into  the  system  test  plan.  Life  tests 
are  time-consuming  and  costly;  therefore, 
life  testing  requirements  and  life  character¬ 
istics  must  be  carefully  analyzed  concur¬ 
rent  with  the  initial  test  design.  Aging  fail¬ 
ure  data  must  be  collected  early  and  ana¬ 
lyzed  throughout  the  testing  cycle.  If  life 
characteristics  are  ignored  until  results  of 
the  test  are  available,  extensive  redesign 
and  project  delays  may  be  required.  Accel¬ 
erated  life  testing  techniques  are  available 
and  may  be  used  whenever  applicable. 

7.4.2  Design  Evaluation/ 

Verification  Testing 

Design  evaluation  and  verification  testing 
is  conducted  by  the  contractor  and/or  the 
development  agency  with  the  primary  ob¬ 
jective  of  influencing  system  design.  De¬ 
sign  evaluation  is  fully  integrated  into  the 
development  test  cycle;  and  its  purposes 
are  to: 

(1)  Determine  if  critical  system  technical 
characteristics  are  achievable; 

(2)  Provide  data  for  refining  and  making 
the  hardware  more  rugged  to  comply  with 
technical  specification  requirements; 


(3)  Eliminate  as  many  technical  and  de¬ 
sign  risks  as  possible  or  to  determine  the 
extent  to  which  they  are  manageable; 

(4)  Provide  for  evolution  of  design  and 
verification  of  the  adequacy  of  design 
changes; 

(5)  Provide  information  in  support  of 
development  efforts; 

(6)  Ensure  components,  subsystems  and 
systems  are  adequately  developed  before 
beginning  operational  tests. 

7.4.3  Design  Limit  Testing 

Design  limit  tests  are  integrated  into  the 
test  program  to  ensure  the  system  will  pro¬ 
vide  adequate  performance  when  operated 
at  outer  performance  limits  and  when  ex¬ 
posed  to  environmental  conditions  ex¬ 
pected  at  the  extreme  of  the  operating  en¬ 
velope.  The  tests  are  based  on  mission  pro¬ 
file  data.  Care  must  be  taken  to  ensure  all 
systems  and  subsystems  are  exposed  to  the 
worst-case  environments,  with  adjustments 
made  because  of  stress  amplification  fac¬ 
tors  and  cooling  problems.  Care  must  also 
be  taken  to  ensure  that  the  system  is  not 
operated  beyond  the  specified  design  lim¬ 
its;  for  example,  an  aircraft  component  may 
have  to  be  tested  at  temperature  extremes 
from  an  Arctic  environment  to  a  desert 
environment. 

7.4.4  Reliability  Development 
Testing  (RDT) 

Reliability  development  testing  (RDT)  or 
reliability  growth  testing  (RGT)  is  a  planned 
test,  analyze,  fix  and  test  (TAFT)  process  in 
which  development  items  are  tested  under 
actual  or  simulated  mission-profile  envi¬ 
ronments  to  disclose  design  deficiencies 
and  to  provide  engineering  information  on 
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failure  modes  and  mechanisms.  The  pur¬ 
pose  of  RDT  is  to  provide  a  basis  for  early 
incorporation  of  corrective  actions  and  veri¬ 
fication  of  their  effectiveness  in  improving 
the  reliability  of  equipment.  Reliability  de¬ 
velopment  testing  is  conducted  under  con¬ 
trolled  conditions  with  simulated  opera¬ 
tional  mission  and  environmental  profiles 
to  determine  design  and  manufacturing 
process  weaknesses.  The  RDT  process  em¬ 
phasizes  reliability  growth  rather  than  a 
numerical  measurement.  Reliability  growth 
during  RDT  is  the  result  of  an  iterative 
design  process  because,  as  the  failures  oc¬ 
cur,  the  problems  are  identified,  solutions 
proposed,  the  redesign  is  accomplished, 
and  the  RDT  continues.  A  substantial  reli¬ 
ability  growth  TAFT  testing  effort  was  con¬ 
ducted  on  the  F-18  DT&E  for  selected  avi¬ 
onics  and  mechanical  systems.  Although 
the  TAFT  effort  added  $100  million  to  the 
Research,  Development^,  Test  and  Evalua¬ 
tion  (RDT&E)  Program,  it  is  estimated  that 
many  times  that  amount  will  be  saved 
through  lower  operational  and  mainte¬ 
nance  costs  throughout  the  system's  life. 

7.4.5  Reliability,  Availability 
and  Maintainability  (RAM) 

Reliability,  availability  and  maintainabil¬ 
ity  (RAM)  requirements  are  assessed  dur¬ 
ing  all  contractor  and  government  testing. 
The  data  are  collected  from  each  test  event 
and  placed  in  a  RAM  data  base,  which  is 
managed  by  the  development  agency.  Con¬ 
tractor  and  government  development  tests 
provide  a  measure  of  the  system's  common 
RAM  performance  against  stated  specifi¬ 
cations  in  a  controlled  environment.  The 
primary  emphasis  of  RAM  data  collection 
during  the  DT  is  to  provide  an  assessment 
of  the  system  RAM  parameters  grow  th  and 
a  basis  for  assessing  the  consequences  of 
any  differences  anticipated  during  field 
operations. 


7.5  SYSTEM  DESIGN  FOR  TESTING 

Built-in  test  (BIT),  built-in-test  equipment 
(BITE)  and  automated  test  equipment  (ATE) 
are  major  areas  that  must  be  considered 
from  the  start  of  the  design  effort.  Design 
for  testing  (Figure  7-2)  addresses  the  need 
to;  (1)  collect  data  during  the  development 
process  concerning  particular  performance 
characteristics;  (2)  enable  efficient  and  eco¬ 
nomical  production  by  providing  ready 
access  to,  and  measurement  of,  appropri¬ 
ate  acceptance  parameters;  and  (3)  enable 
rapid  and  accurate  assessment  of  the  status 
of  the  product  to  the  lowest  repairable  ele¬ 
ment  when  deployed.  Many  hardware  sys¬ 
tems  have  testing  circuits  designed  and 
built-in.  This  early  planning  by  design  en¬ 
gineers  allows  easy  testing  for  fault  isola¬ 
tion  of  circuits,  both  in  system  develop¬ 
ment  phases  and  during  operational  test¬ 
ing  and  deployment  There  are  computer 
chips  in  which  more  than  half  of  the  circuits 
are  for  test/circuit  check  functions.  This 
type  of  circuit  design  requires  early  plan¬ 
ning  by  the  PM  to  ensure  the  RFP  require¬ 
ments  include  the  requirement  for  de¬ 
signed/built-in  test  capability.  Evaluation 
o?  these  BIT/BITE/ ATE  systems  must  be 
included  in  the  test  program. 

7.6  IMPACT  OF 
WARRANTIES  ON  T&E 

A  warranty  or  guarantee  is  a  commitment 
provided  by  a  supplier  to  deliver  a  product 
that  meets  specified  standards  for  a  speci¬ 
fied  time.  With  a  properly  structured  war¬ 
ranty,  the  contractor  must  meet  technical 
and  operational  requirements.  If  the  prod¬ 
uct  should  fail  during  that  warranty  pe¬ 
riod,  the  contractor  must  replace  or  make 
repairs  at  no  additional  cost  to  the  govern¬ 
ment.  The  Defense  Appropriations  Act  of 
1984  requires  warranties  or  guarantees  on 
all  weapon  systems  procurement.  This  act 
makes  warranties  a  standard  item  on  most 
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Figure  7-2.  Design  For  Testing;  Procedures 


fixed-price  production  contracts.  Incentives 
are  the  main  thrust  of  warranties;  and,  as 
prescribed  in  MIL-STD-781,  the  govern¬ 
ment  will  perform  a  reliability  demonstra¬ 
tion  test  on  the  system  to  determine  these 
incentives.  Although  warranties  have  fa¬ 
vorable  advantages  to  the  government  dur¬ 
ing  the  early  years  of  the  contract,  warran¬ 
ties  do  not  uix.'Ct  the  types  of  testing  per¬ 
formed  to  ensure  the  system  meets  techni¬ 
cal  specifications  and  satisfies  operational 
effectiveness  and  suitability  requirements. 
Warranties  do,  however,  affect  the  amount 
of  testing  required  lo  establish  reliability. 
Because  the  standard  item  is  warraii^ed, 
less  emphasis  on  that  portion  of  the  item 
can  allow  for  additional  emphasis  on  other 
aspects  of  the  item  not  covered  under  the 
warranty.  Further,  the  government  may 
tend  to  have  more  confidence  in  contractor 
test  results  and  may  be  able,  therefore,  to 
avoid  some  duplication  of  test  effort.  The 
warranty  essentially  shifts  the  burden  of 
risk  from  the  government  to  the  contractor. 
Warranties  can  significantly  increase  the 
price  of  the  contract,  especially  if  high-cost 
components  are  involved. 

7.7  DT&E  OF  LIMITED 
PROCUREMENT  QUANTITY 
PROGRAMS 

Programs  that  involve  .he  procurement  of 
relatively  few  items,  typrally  over  an  ex¬ 


tended  period,  are  normally  subjected  to 
standard  DT&E.  Occasionally,  a  unique  test 
approach  that  deviates  from  the  standard 
timing  and  reporting  schedu  le  will  be  used . 
The  DT&E  principle  of  components,  sub¬ 
systems,  prototypes  and  first-production 
models  of  the  system  is  normally  applied  to 
limited  procurement.  It  is  important  that 
DT&E  and  OT&E  organizations  work  to¬ 
gether  to  ensure  that  T&E  plans  are  inte¬ 
grated  into  the  overall  acquisition  strategy. 

7.8  SUMMARY 

Development  test  and  evaluation  is  an  it¬ 
erative  process  of  designing,  building,  test¬ 
ing,  identifying  deficiencies,  fixing,  retest¬ 
ing  and  repeating.  It  is  performed  in  the 
factory,  laboratory  and  on  the  proving 
ground  by  the  contractors  and  the  govern¬ 
ment.  Contractor  and  government  testing 
is  combined  into  one  integrated  test  pro¬ 
gram  and  conducted  to  determine  if  the 
technical  development  of  the  acquisition 
process  have  been  met  and  to  provide  data 
to  the  decision  authority. 
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8 

DT&E  SUPPORT  OF  TECHNICAL  REVIEWS 
AND  MILESTONE  DECISIONS 


8.1  INTRODUCTION 

Throughout  the  acquisition  process,  devel¬ 
opment  test  and  evaluation  (DT&E)  is  ori¬ 
ented  toward  the  demonstration  of  specifi¬ 
cations  showing  the  completeness  and  ad¬ 
equacy  of  systems  engineering,  design, 
development  and  performance.  A  critical 
purpose  of  DT&E  is  to  identify  the  risks  of 
development  by  testing  and  evaluating  se¬ 
lected  high-risk  components  or  subsystems. 
Development  test  and  evaluation  is  the 
developer's  tool  to  show  that  the  system 
performs  as  specified  or  that  deficiencies 
have  been  corrected  and  the  system  is  ready 
for  operational  testing  and  fielding.  The 
DT&E  results  are  used  throughout  the  sys¬ 
tem  engineering  process  to  provide  valu¬ 
able  data  in  support  of  formal  design  re¬ 
views.  This  chapter  describes  the  types  of 
development  testing  used  throughout  the 
system  acquisition  cycle  to  support  the 
materiel  acquisition  process.  It  also  de¬ 
scribes  the  objectives  of  the  various  tests 
conducted  during  the  DT&E  process  and 
discusses  their  relationship  to  the  formal 
design  reviews  essential  to  the  systems 
engineering  process. 

8.2  DT&E  AND  THE  SYSTEM 
ACQUISITION  CYCLE 

As  illustrated  in  Figure  8-1,  DT&E  is  con¬ 
ducted  throughout  the  system  life  cycle. 
Development  test  and  evaluation  may  be¬ 
gin  before  program  initiation  (Milestone  0) 
with  the  evaluation  of  evolving  technol¬ 


ogy,  and  it  continues  after  the  system  is  in 
the  field. 

8.2.1  DT&E  Prior  to  Program  Initiation 

Prior  to  program  initiation,  technology  fea¬ 
sibility  testing  is  conducted  to  confirm  that 
the  technology  considered  for  the  proposed 
weapon  development  is  the  most  advanced 
available  and  that  it  is  technically  feasible. 

8.2.2  DT&E  During  the  Concept 
Exploration  and  Definition  Phase 

Development  testing  that  takes  place  dur¬ 
ing  the  Concept  Exploration  and  Defini¬ 
tion  Phase  is  conducted  by  a  contractor  or 
the  government  to  assist  in  selecting  pre¬ 
ferred  alternative  system  concepts,  tech¬ 
nologies  and  designs.  The  testing  conducted 
depends  on  the  state  of  development  of  the 
test  article's  design.  Government  test  evalu¬ 
ators  participate  in  this  testing  because  in¬ 
formation  obtained  can  be  used  to  support 
the  Systems  Requirements  Review.  The  in¬ 
formation  obtained  from  these  tests  may 
also  be  used  to  support  a  program  start 
decision  by  the  Services  or  the  Office  of  the 
Secretary  of  Defense  (OSD). 

8.2.3  DT&E  During  the  Demonstration 
and  Validation  Phase 

Development  testing  conducted  during  the 
Demonstration  and  Validation  Phase  is 
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used  to  demonstrate  that:  technical  risk 
areas  have  been  identified  and  can  be  re¬ 
duced  to  acceptable  levels;  the  best  techni¬ 
cal  approach  can  be  identified;  and,  from 
this  point  on,  engineering  efforts  will  be 
required  rather  than  experimental  efforts. 
It  supports  the  Milestone  II  decision  that 
considers  entry  into  Engineering  and  Manu¬ 
facturing  Development  (EMD)  and,  as  ap¬ 
propriate,  low-rate  initial  production.  This 
DT&E  includes  contractor/government 
integrated  testing,  engineering  design  test¬ 
ing  and  advanced  development  verifica¬ 
tion  testing. 

Development  testing  during  this  period  is 
most  often  conducted  at  the  contractor's 
facility.  It  is  conducted  on  components, 
subsystems,  brassboard  configurations  or 
advanced  development  prototypes  to 
evaluate  the  potential  application  of  tech¬ 
nology  and  related  design  approaches  be¬ 
fore  EMD.  Component  interface  problems 
and  equipment  performance  capabilities 
are  evaluated.  The  use  of  properly  vali¬ 
dated  analysis,  modeling  and  simulation  is 
encouraged,  especially  during  the  early 
phases  to  assess  those  areas  that,  for  safety 
or  testing  capability  limitations,  cannot  be 
observed  directly  through  testing.  Models 
can  provide  early  projections  of  systems 
performance,  effectiveness  and  suitability 
and  can  reduce  testing  costs.  This  test  and 
evaluation  also  may  include  initial  envi¬ 
ronmental  assessments. 

Army  testing  of  the  Advanced  Attack  He¬ 
licopter  ( AAH)  provides  an  example  of  the 
type  of  activities  that  occur  during  devel¬ 
opment  tests  (DTs).  The  early  DT&E  of  the 
AAH  was  conducted  by  the  Army  Engi¬ 
neering  Flight  Activity.  The  test  was  con¬ 
ducted  in  conjunction  with  an  Early  Opera¬ 
tional  Test,  and  candidate  designs  were 
flown  more  than  90  hours  to  evaluate  flight 
handling  qualities  and  aircraft  performance. 


This  test  also  included  the  firing  of  the  30 
millimeter  cannon  and  the  2.75-inch  rock¬ 
ets.  Reliability,  availability  and  maintain¬ 
ability  (RAM)  data  were  obtained  through¬ 
out  the  test  program;  and  these  data,  along 
with  RAM  data  provided  from  early  con¬ 
tractor  testing,  became  a  part  of  the  system's 
RAM  data  base.  After  evaluating  the  re¬ 
sults,  the  Army  selected  a  contractor  to 
proceed  with  th?  next  development  phase 
of  the  AAH. 

8.2.4  DT&E  During  the  Engineering 
and  Manufacturing  Development  Phase 

Development  test  and  evaluation  con¬ 
ducted  during  the  EMD  Phase  provides  the 
final  technical  data  for  determining  a 
system's  readiness  to  transition  into  either 
low-rate  initial  production  (TRIP)  or  full- 
rate  production.  It  is  conducted  using  ad¬ 
vanced  engineering  development  models 
and  is  characterized  by  engineering  and 
scientific  approaches  under  controlled  con¬ 
ditions.  The  qualification  testing  provides 
quantitative  and  qualitative  data  for  use  in 
the  system's  evaluation.  The  evaluation  re¬ 
sults  are  used  by  the  development  commu¬ 
nity  and  are  also  provided  to  Service  and 
OSD  decision  authorities.  These  tests  mea¬ 
sure  technical  performance  including:  ef¬ 
fectiveness,  reliability,  availability,  main¬ 
tainability,  compatibility,  interoperability, 
safety  and  supportability .  They  include  tests 
of  human  engineering  and  technical  as¬ 
pects  of  the  system.  Demonstrations  of 
whether  engineering  is  reasonably  com¬ 
plete  and  if  solutions  to  all  significant  de¬ 
sign  problems  are  in  hand  are  also  included. 
Development  test  and  evaluation  may  be 
conducted  on  LRIP  articles  as  a  prelude 
certifying  the  system  ready  for  initial  op¬ 
erational  test  and  evaluation  (lOT&E).  The 
Navy  calls  this  DT&E  for  certification 
TECHEVAL  (technical  evaluation). 
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As  an  example  of  testing  done  during  the 
EMD  Phase,  the  Army  AAH  was  flown  in 
a  series  of  engineering  design  tests  (EDTs). 
The  EDT-1,  -2  and  -4  were  flown  at  the 
contractor's  facility.  (The  EDT-3  require¬ 
ment  was  deleted  during  program  restruc¬ 
turing.)  The  objectives  of  these  flight  tests 
were  to  evaluate  the  handling  characteris¬ 
tics  of  the  aircraft,  check  significant  perfor¬ 
mance  parameters  and  confirm  the  correc¬ 
tion  of  deficiencies  noted  during  earlier 
testing.  The  EDT-5  was  conducted  at  an 
Army  test  facility,  Yuma  Proving  Ground. 
The  objectives  of  this  test  were  the  same  as 
earlier  EDTs;  however,  the  testers  were 
required  to  ensure  that  all  discrepancies 
were  resolved  before  the  aircraft  entered 
operational  testing.  During  the  EDTs,  op¬ 
erational  test  personnel  were  completing 
operational  test  design,  bringing  together 
test  resources  and  observing  the  DT&E  tests. 
Additionally,  operational  test  (OT)  person¬ 
nel  were  compiling  test  data,  such  as  the 
system  contractor's  test  results,  from  other 
sources.  The  evolving  DT  results  and  con¬ 
tractor  data  were  made  available  to  the 
Critical  Design  Review  members  to  ensure 
that  each  configuration  item  design  was 
essentially  completed.  The  Army  conducted 
a  physical  Configuration  Audit  to  provide 
a  technical  examination  to  verify  that  each 
item  "as  built"  conformed  to  the  technical 
documentation  defining  that  item. 

8.2.5  DT&E  During  the  Production 
and  Deployment  Phase 

Development  testing  may  be  conducted 
after  the  TRIP  Phase  and  after  the  full-rate 
production  decision  is  made  at  Milestone 
III.  This  testing  is  normally  tailored  to  verify 
correction  of  identified  design  problems 
and  demonstrate  the  system  modification's 
readiness  for  production.  This  testing  is 
conducted  under  controlled  conditions  and 
provides  quantitative  and  qualitative  data. 
This  testing  is  conducted  on  production 
items  delivered  from  either  the  pilot  or 
initial  production  runs.  To  ensure  that  the 


items  are  produced  according  to  contract 
specification,  quantity  production  pro¬ 
cesses  are  used.  This  testing  determines 
whether  the  system  has  successfully 
transitioned  from  engineering  development 
prototype  to  production  and  whether  it 
meets  design  specifications. 

8.2.6  DT&E  During  Operations 
and  Support 

The  development  testing,  which  occurs 
soon  after  the  initial  operating  capability  or 
initial  deployment,  assesses  the  deployed 
system's  operational  readiness  and  sup- 
portability.  It  ensures  that  all  deficiencies 
noted  during  previous  testing  have  been 
corrected,  evaluates  proposed  product  im¬ 
provements  and  block  upgrades,  and  en¬ 
sures  that  integrated  logistics  support  is 
complete.  It  also  evaluates  the  resources  on 
hand  and  determines  if  the  plans  to  ensure 
operational  phase  readiness  and  support 
objectives  are  sufficient  to  maintain  the 
system  for  the  remainder  of  its  acquisition 
life  cycle.  Near  the  end  of  the  system's  life, 
DT&E  is  performed  to  assist  in  modifying 
the  system  to  help  meet  new  threats,  add 
new  technologies,  or  aid  in  disposal. 

Once  a  system  approaches  the  end  of  its 
usefulness,  the  development  testing  con¬ 
ducted  is  concerned  with  the  monitoring  of 
a  system's  current  state  of  operational  ef¬ 
fectiveness,  suitability  and  readiness  to 
determine  whether  major  upgrades  are  nec¬ 
essary  or  deficiencies  warrant  consider¬ 
ation  of  a  new  system  replacement.  Tests 
are  normally  conducted  by  the  operational 
testing  community;  however,  the  DT&E 
community  may  be  required  to  assess  the 
technical  aspects  of  the  system. 

8.3  DT&E  AND  THE  REVIEW  PROCESS 

8.3.1  The  Technical  Review  Process 

Technical  reviews  and  audits  are  conducted 
by  the  government  and  the  contractor  as 
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part  of  the  system  engineering  process  to 
ensure  the  design  meets  the  system,  sub¬ 
system  and  software  specifications.  Each 
review  is  unique  in  its  timing  and  orienta¬ 
tion.  Some  reviews  build  on  previous  re¬ 
views  and  take  the  design  and  testing  effort 
one  step  closer  to  the  final  system  design  to 
satisfy  the  operational  concept/purpose  for 
the  weapon  system.  Table  8-1  illustrates 
the  sequencing  of  the  technical  reviews  in 
relation  to  the  test  and  evaluation  phases. 

The  review  process  was  established  to  en¬ 
sure  that  the  system  under  development 
would  meet  government  requirements.  The 
reviews  evaluate  data  from  contractor  and 
govermnent  testing,  engineering  analysis, 
and  models  to  determine  if  the  system  or  its 
components  will  eventually  meet  all  func¬ 
tional  and  physical  specifications  and  to 
determine  the  final  system  design.  The  sys¬ 
tem  specification  is  very  important  in  this 
process.  It  is  the  document  used  as  a  bench 
mark  to  compare  contractor  progress  in 
designing  and  developing  the  desired  prod¬ 
uct.  The  requirements  and  direction  for 
these  formal  technical  reviews  and  audits 
are  set  forth  in  MIL-STD-1521B. 

8.3.2  Testing  in  Support  of 
Technical  Reviews 

The  testing  community  must  be  continu¬ 
ally  involved  in  supporting  the  technical 
reviews  of  their  systems.  Decisions  made  at 
these  reviews  have  major  impacts  on  the 
system  test  design,  resources  required  to 
test,  and  the  development  of  the  Test  and 
Evaluation  Master  Plan  (TEMP)  and  other 
documentation.  A  more  detailed  discus¬ 
sion  of  testing  to  support  the  technical  re¬ 
views  is  provided  in  the  Systems  Engineer¬ 
ing  Management  Guide  (Reference  45) .  The 
reviews  focus  primarily  on  government 
technical  specifications  for  the  system.  Fig¬ 
ure  8-2  illustrates  the  program  specifica¬ 


tions  and  how  they  are  developed  in  the 
system  life  cycle. 

8.3.3  Formal  Reviews 

8.3.3.1  Systems  Requirements 
Review  (SRR) 

The  SRR  is  normally  conducted  during  the 
System  Concept  Exploration  and  Defini¬ 
tion  Phase  or  early  Demonstration  and 
Validation  Phase.  It  consists  of  a  review  of 
the  system/  system  segment  specifications, 
also  known  as  the  "A"  specifications  (Sys¬ 
tem  Functional  Block  Diagram,  Reference 
45,  Chapter  12),  and  is  conducted  after  the 
accomplishment  of  functional  analysis  and 
preliminary  requirements  allocation.  Dur¬ 
ing  this  review,  the  systems  engineering 
management  activity  and  its  output  are 
reviewed  for  responsiveness  to  the  State¬ 
ment  of  Work  requirements.  The  primary 
function  of  the  SRR  is  to  ensure  that  sys¬ 
tems  requirements  have  been  completed 
and  properly  identified  and  that  there  is  a 
mutual  understanding  between  the  con¬ 
tractor  and  the  government.  During  the 
review,  the  contractor  describes  his  progress 
and  any  problems  in  risk  identification  and 
ranking,  risk  avoidance  and  reduction, 
crade-off  analysis,  producibility  and  manu¬ 
facturing  considerations,  and  hazards  con¬ 
siderations.  The  results  of  integrated  test 
planning  are  reviewed  to  ensure  the  ad¬ 
equacy  of  planning  to  assess  the  design  and 
to  identify  risks.  Issues  of  testability  of  re¬ 
quirements  should  be  discussed. 

8.3.3.2  Systems  Design  Review  (SDR) 

The  SDR  is  conducted  as  a  final  review 
before  submittal  of  the  Demonstration  and 
Validation  Phase  products  or  as  the  initial 
EMD  Phase  review.  The  "A"  specification 
is  validated  to  ensure  that  the  most  current 
specification  is  included  in  the  System  Func¬ 
tional  Baseline  and  that  they  are  adequate 
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Table  8-1 .  Technical  Reviews  and  Audits 


WHEN 

PURPOSE 

DOCUMENTATION/ 

DATA 

System 

Requirements 

Review 

SRR 

Late 

C/E 

-  Evaluate  System 

Functional  Requirements 

-  Prelim  ■A”  Spec 

-  Prelim  Planning 
Documentation 

-  FFBD,  RAS,  MBN 
Analysis 

System 

Design 

Review 

SDR 

Late 

Dem 

Val 

-  Evaluate  System  Design 

-  Validate  “A"  Spec 

-  Establish  System 

Level  Functional  Baseline 

-  -A"  Spec 

-  Prelim  "B"  Spec 

-  Design  Documents 

-  RAS,  SSD,  TLS 

Software 

Specification 

Review 

SSR 

Early 

EMD 

Prior 

SWPDR 

-  Evaluate  SW 

Performance  Requirements 

-  Validate  "B-6“  Specs 

-  Establish  SW  Specs 

Baseline 

-  “B-5“  Spec 

-  (SRS  &  IRS) 

-  Ops  Concept  Doc 

Preliminary 

Design 

Review 

PDR 

Early 

EMD 

-  Validate  "B"  Specs 

-  Establish  HW  Allocated 
Baseline 

-  Evaluate  Preliminary 

Design  HW  &  SW 

-  “B“  Spec 

-  Des  Doc  Test  Plan 

-  ICD,  Engr  Drawings 

-  Preliminary  SDD  -  IDD 

Critical 

Design 

Review 

CDR 

Early/Mid 

EMD 

-  Evaluate  Cl  Design 

-  Determine  Readiness 

For  Fabrication 

•  Prelim  C,  D,  E  Specs 

•  Detail  Design 

Documents  Include 

SDD  -  IDD 

Test 

Readiness 

Review 

TRR 

Mid/Late 

EMD 

-  Approve  SW  Test 

Procedures 
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Figure  8-2.  Specifications  Summary 


and  cost-effective  to  satisfy  validated  mis¬ 
sion  requirements.  The  SDR  encompasses 
the  total  system  requirement  of  operations, 
maintenance,  test,  training,  computers,  fa¬ 
cilities,  personnel  and  logistics  consider¬ 
ations.  A  technical  understanding  should 
be  reached  on  the  validity  and  the  degree  of 
completeness  of  specifications,  design,  op¬ 
erational  concept  documentation,  software 
requirements  specifications  and  interface 
requirements  specifications  during  this  re¬ 
view. 

8.3.3.3  Software  Specification 
Review  (SSR) 

The  SSR  is  a  formal  review  of  the  computer 
system  configuration  item  (CSCI)  require¬ 
ments,  normally  held  after  a  SDR  but  be¬ 
fore  the  start  of  a  CSCI  preliminary  design- 
Its  purpose  is  to  validate  the  allocated 
baseline  for  preliminary  CSCI  design  by 
demonstrating  to  the  government  the  ad¬ 
equacy  of  the  software  requirements  speci¬ 
fications,  interface  requirements  specifica¬ 
tions  and  operational  concept  documenta¬ 
tion. 

8.3.3.4  Preliminary  Design 
Review  (PDR) 

The  PDR  is  a  formal  technical  review  of  the 
basic  approach  for  a  configuration  item.  It 
is  conducted  at  the  configuration  item  and 
system  level  early  in  the  HMD  Phase  to 
confirm  that  the  preliminary  design  logi¬ 
cally  follows  SDR  findings  and  meets  the 
system  requirements.  The  review  results  in 
an  approval  to  begin  the  detailed  design. 
The  draft  Type  B  specifications  are  reviewed 
during  the  PDR.  The  purpose  of  the  PDR  is 
to:  evaluate  the  progress,  technical  ad¬ 
equacy  and  risk  resolution  (on  technical, 
cost  and  schedule  basis)  of  the  configura¬ 
tion  item  design  approach;  conduct  DT  and 
OT  activities  to  measure  the  performance 
of  each  configuration  item  (Cl);  and  estab¬ 


lish  the  existence  and  compatibility  of  the 
physical  and  functional  interface  among 
the  Cl  and  other  equipment. 

8.3.3.5  Critical  Design  Review  (CDR) 

The  CDR  may  be  conducted  on  each  con¬ 
figuration  item  and/or  at  the  system  level. 
It  is  conducted  during  the  EMD  Phase  when 
the  detailed  design  is  essentially  complete 
and  prior  to  the  Functional  Configuration 
Audit.  During  the  CDR,  the  overall  techni¬ 
cal  program  risks  associated  with  each  con¬ 
figuration  item  are  also  reviewed  on  a  tech¬ 
nical,  cost  and  schedule  basis.  It  includes  a 
review  of  the  "C"  specifications  and  the 
status  of  both  the  system's  hardware  and 
software.  Input  from  qualification  testing 
should  assist  in  determination  of  readiness 
for  design  freeze  and  LRIP. 

8.3.4.6  Test  Readiness  Review  (TRR) 

The  TRR  is  a  formal  review  of  the 
contractor's  readiness  to  begin  CSCI  test¬ 
ing.  A  government  witness  will  observe  the 
system  demonstration  to  verify  that  the 
system  is  ready  to  proceed  with  CSCI  test¬ 
ing.  It  is  conducted  after  the  software  test 
procedures  are  available  and  computer 
software  components  testing  is  complete. 
The  purpose  of  the  TRR  is  for  the  program 
management  office  (PMO)  to  determine 
whether  the  contractor  is  ready  to  begin 
CSCI  testing. 

8.3.4.7  Functional  Configuration 
Audit  (FCA) 

The  FCA  is  a  formal  review  to  verify  that 
the  configuration  item's  performance  com¬ 
plied  with  its  development  specification. 
The  "B"  specifications  are  derived  from  the 
system  requirements  and  baseline  docu¬ 
mentation.  During  the  FCA,  all  relevant 
test  data  is  reviewed  to  verify  that  the  item 
has  performed  as  required  by  its  functional 
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and/or  allocated  configuration  identifica¬ 
tion.  The  audit  is  conducted  on  the  item 
representative  (prototype  or  production) 
of  the  configuration  to  be  released  for  pro¬ 
duction.  The  audit  consists  of  a  review  of 
the  contractor's  test  procedures  and  re¬ 
sults.  Information  provided  will  be  used  by 
the  FC  A  to  determine  the  status  of  planned 
tests. 

8.3.3.9  Physical  Configuration 
Audit  (PCA) 

The  PCA  is  a  formal  review  which  estab¬ 
lishes  the  product  baseline  as  reflected  in 
an  early  production  configuration  item.  It 
is  the  examination  of  the  as-built  version  of 
hardware  and  software  configuration  items 
against  its  technical  documentation.  The 
PCA  also  determines  that  the  acceptance 
testing  requirements  prescribed  by  the 
documentation  are  adequate  for  accep¬ 
tance  of  production  units  of  a  Cl  by  quality 
assurance  activities.  It  includes  a  detailed 
audit  of  engineering  drawings,  final  Part  II 
product  specifications,  technical  data  and 
plans  for  testing  that  will  be  utilized  during 
production.  The  PCA  is  performed  on  all 
first  articles  and  on  the  first  CIs  delivered 
by  a  new  contractor. 

8.3.3.9  Formal  Qualification 
Review  (FQR) 

The  FQR  is  a  systems-level  configuration 
audit  that  may  be  conducted  after  system 
testing  is  completed.  The  objective  is  to 
verify  that  the  actual  performance  of  the  Cl, 
as  determined  through  testing,  complies 
with  its  Type  B  specifications  and  to  docu¬ 
ment  the  results  of  the  qualification  tests. 
The  FQR  and  FCA  are  often  performed  at 
the  same  time;  however,  if  sufficient  test 
results  are  not  available  at  the  FCA  to  en¬ 
sure  the  Cl  will  perform  in  its  operational 
environment,  the  FQR  can  be  scheduled  for 
a  later  time. 


8.3.3.10  Production  Readiness 
Review  (PRR) 

The  PRR  is  an  assessment  of  the  contractor's 
ability  to  produce  the  items  on  the  contract. 
It  is  usually  a  series  of  reviews  conducted 
before  an  TRIP  or  full-scale  production 
decision.  For  more  information,  see  Chap¬ 
ter  10,  paragraph  10.3  (Production  Readi¬ 
ness  Reviews). 

8.3.3.11  Configuration  Change  Control 

The  Configuration  Change  Control  review 
is  an  assessment  of  the  impact  of  engineer¬ 
ing  or  design  changes.  It  is  conducted  by 
the  engineering,  T&E  and  PM  portions  of 
the  PMO.  Most  approved  Class  I  engineer¬ 
ing  change  proposals  will  require  addi¬ 
tional  testing,  and  the  test  manager  must 
accommodate  the  new  schedules  and  re¬ 
source  requirements.  Adequate  testing 
must  be  accomplished  to  ensure  integra¬ 
tion  and  compatibility  of  these  changes. 
For  example,  an  engineering  change  re¬ 
view  was  conducted  to  replace  the  black 
and  white  monitors  and  integrate  color 
monitors  into  the  Airborne  Warning  and 
Control  System  (AWACS).  Further,  the 
AW  ACS  operating  software  had  to  be  up¬ 
graded  to  handle  color  enhancement.  The 
review  was  conducted  by  the  government 
PMO;  and  sections  of  the  PMO  were  tasked 
to  contract,  test,  engineer,  logistically  sup¬ 
port,  control,  cost  and  finance  the  change  to 
completion.  Configuration  control  and  en¬ 
gineering  changes  are  discussed  in  MIL- 
STD-480. 

8.4  SbMMARY 

Design  reviews  are  an  integral  and  essen¬ 
tial  part  of  the  system  engineering  process. 
The  meetings  range  from  very  formal  re¬ 
views  by  government  and  contractor  PMs 
to  informal  technical  reviews  concerned 
with  product  or  task  elements  of  the  work 
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breakdown  structure.  Reviews  may  be  con¬ 
ducted  in  increments  over  time.  All  re¬ 
views  share  the  common  objective  of  deter¬ 
mining  the  technical  adequacy  of  the  exist¬ 
ing  design  to  meet  technical  requirements. 


The  DT/OT  assessments  and  test  results 
are  made  available  to  the  reviews,  and  it  is 
important  that  the  test  community  be  in¬ 
volved. 
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9 

COMBINED  AND  CONCURRENT 
TESTING 


9.1  INTRODUCTION 

The  terms  "concurrency,"  "concurrent  test¬ 
ing,"  and  "combined  testing"  are  some¬ 
times  subject  to  misinterpretation. 
Concurrency  is  defined  as  an  approach  to 
system  development  and  acquisition  in 
which  phases  of  the  acquisition  process, 
which  normally  occur  sequentially,  over¬ 
lap  to  some  extent.  For  example,  a  weapon 
system  enters  the  production  phase  while 
development  efforts  are  still  underway. 

Concurrent  testing  refers  to  circumstances 
when  development  testing  and  operational 
testing  take  place  at  the  same  time  as  two 
parallel  but  separate  and  distinct  activities. 
In  contrast,  combined  testing  refers  to  a 
single  test  program  conducted  to  support 
development  test  (DT)  and  operational  test 
(OT)  objectives.  This  chapter  discusses  the 
use  of  combined  testing  and  concurrent 
testing,  and  highlights  some  of  the  advan¬ 
tages  and  disadvantages  associated  with 
these  approaches. 

9.2  COMBINING  DEVELOPMENT 
TEST  AND  OPERATIONAL  TEST 

Certain  test  events  can  be  organized  to 
provide  information  useful  to  development 
testers  and  operational  testers.  For  example, 
a  prototype  free-fall  munitions  could  be 
released  from  a  fighter  aircraft  at  opera¬ 
tional  employment  conditions  instead  of 
from  a  static  stand  to  satisfy  DT  and  OT 
objectives.  Such  instances  need  to  be  iden¬ 
tified  to  prevent  unnecessary  duplication 
of  effort  and  to  control  costs.  A  combined 


testing  approach  is  also  appropriate  for 
certain  specialized  types  of  testing.  For  ex¬ 
ample,  in  the  case  of  nuclear  survivability 
and  hardness  testing,  systems  cannot  be 
tested  in  a  totally  realistic  operational  envi¬ 
ronment;  therefore,  a  single  test  program  is 
often  used  to  meet  both  development  and 
operational  test  objectives. 

The  DODI  5000.2  encourages  combined 
testing  and  states  that  "a  combined  DT&E 
[development  test  and  evaluation]  and 
OT&E  [operational  test  and  evaluation] 
approach  should  be  considered  when  there 
are  time  and  cost  savings.  The  combined 
approach  must  not  compromise  either  DT 
or  OT  objectives."  If  this  approach  is  elected, 
planning  efforts  must  be  carefully  coordi¬ 
nated  early  in  the  program  to  ensure  data  is 
obtained  to  satisfy  the  needs  of  both  the 
developing  agency  and  the  independent 
operational  tester.  Care  must  also  be  exer¬ 
cised  to  ensure  a  combined  test  program 
contains  dedicated  operational  test  events 
to  satisfy  the  requirement  for  an  indepen¬ 
dent  evaluation.  A  final  independent  phase 
of  OT&E  testing  shall  be  required  for  be¬ 
yond  low-rate  initial  production  (BLRIP) 
decisions.  In  all  combined  test  programs, 
provisions  for  separate  independent  de¬ 
velopment  and  operational  evaluations  of 
test  results  should  be  provided. 

Service  regulations  describe  the  sequence 
of  activities  in  a  combined  testing  program 
as  follows: 
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Although  OT&E  is  separate  and  dis¬ 
tinct  from  DT&E,  most  of  the  gener¬ 
ated  data  are  mutually  beneficial  and 
freely  shared.  Similarly,  the  resources 
needed  to  conduct  and  support  both 
test  efforts  are  often  the  same  or  very 
similar.  Thus,  when  sequential  DT&E 
and  OT&E  efforts  would  cause  delay 
or  increase  the  acquisition  cost  of  the 
system,  DT&E  and  OT&E  are  com¬ 
bined.  When  combined  testing  is 
planned,  the  necessary  test  conditions 
and  data  required  by  both  DT&E  and 
OT&E  organizations  must  be  inte¬ 
grated.  Combined  testing  can  normally 
be  divided  into  three  segments.  In  the 
first  segment,  DT&E  event[s]  usually 
assume  priority  because  critical  tech¬ 
nical  and  engineering  tests  must  be 
accomplished  to  continue  the  engineer¬ 
ing  and  development  process.  During 
this  early  period,  OT&E  personnel 
participate  to  gain  familiarity  with  the 
system  and  to  gain  access  to  any  test 
data  that  can  support  OT&E.  Next,  the 
combined  portion  of  the  testing  fre¬ 
quently  includes  shared  objectives  or 
joint  data  requirements.  The  last  seg¬ 
ment  normally  contains  the  dedicated 
OT&E  or  separate  OT&E  events  to  be 
conducted  by  the  OT&E  agency.  The 
OT &E  agency  and  implementing  com¬ 
mand  must  ensure  the  combined  test 
is  planned  and  executed  to  provide  the 
necessary  operational  test  information. 
The  OT&E  agency  provides  an  inde¬ 
pendent  evaluation  of  the  OT&E  por¬ 
tion  and  is  ultimately  responsible  for 
achieving  OT&E  objectives. 

The  testing  of  the  Navy's  F-14  aircraft  has 
been  cited  as  an  example  of  a  successful 
combined  test  and  evaluation  (T&E)  pro¬ 
gram  (Reference  112).  A  key  factor  in  the 
success  of  the  F-14  approach  was  the  selec¬ 
tion  of  a  T&E  coordinator  responsible  for 
supervising  the  generation  of  test  plans 


that  integrated  the  technical  requirements 
of  the  developers  with  the  operational  re¬ 
quirements  of  the  users.  The  T&E  coordi¬ 
nator  was  also  responsible  for  the  alloca¬ 
tion  of  test  resources  and  the  overall  man¬ 
agement  of  the  test.  In  a  paper  for  the 
Defense  Systems  Management  College,  Mr. 
Thomas  Hoivik  describes  the  successful 
F-14  test  program  as  follows: 

The  majority  of  the  Navy  develop¬ 
mental  and  operational  testing  took 
place  during  the  same  period  and  even 
on  the  same  flights.  Maximum  use  was 
made  of  contractor  demonstrations 
witnessed  by  the  Navy  testing  activi¬ 
ties  to  obviate  the  retesting  of  a  techni¬ 
cal  point  already  demonstrated  by  the 
contractor.  Witnessing  by  testing  ac¬ 
tivities  was  crucially  important  and 
allowed  the  contractor's  data  to  be 
readily  accepted  by  the  testing  activi¬ 
ties.  This  approach  also  helped  to  elimi- 
nate  redundancy  in  testing,  i.e.  the 
testing  of  the  same  performance  pa¬ 
rameter  by  several  different  activities 
which  has  been  a  consistent  and  waste¬ 
ful  feature  of  Navy  testing  in  the  past. 

Obviously,  this  approach  placed  a  great 
deal  of  responsibility  directly  on  the  shoul¬ 
ders  of  the  T&E  Coordinator,  and  required 
his  staff  to  deal  knowledgeably  with  a  wide- 
ranging  and  complex  test  plan. 

9.3  CONCURRENT  TESTING 

In  1983,  a  senior  DOD  test  and  evaluation 
official  testified  that  a  concurrent  testing 
approach  is  usually  not  an  effective  strat¬ 
egy  (Reference  106).  He  acknowledged, 
however,  that  certain  test  events  may  pro¬ 
vide  information  useful  to  development 
and  operational  testers,  and  test  plarmers 
must  be  alert  to  identify  those  events.  His 
testimony  included  the  following  examples 
of  situations  where  a  concurrent  testing 
approach  was  unsuccessful: 


9-2 


(1)  During  AAH  (Advanced  Attack 
Helicopter)  testing  in  1981,  the  Target 
Acquisition  Designation  System 
(TADS)  was  undergoing  developmen¬ 
tal  and  operational  testing  at  the  same 
time.  The  schedule  did  not  allow 
enough  time  for  qualification  testing 
(a  development  test  activity)  of  the 
TADS  prototype  prior  to  a  full  field 
test  of  the  total  aircraft  system,  nor  was 
there  time  to  introduce  changes  to 
TADS  problems  discovered  in  tests. 
As  a  result,  the  T  ADS  performed  poorly 
and  was  unreliable  during  the  opera¬ 
tional  test.  The  resulting  DSARC  [De¬ 
fense  Systems  Acquisition  Review 
Council]  action  required  the  Army  to 
fix  and  retest  the  TADS  prior  to  release 
of  second  year  and  subsequent  pro¬ 
duction  funds. 

(2)  When  the  AIM-7  Sparrow  air-to- 
air  missile  was  tested  an  attempt  was 
made  to  move  into  operational  testing 
while  developmental  reliability  test¬ 
ing  was  still  underway.  The  opera¬ 
tional  test  was  suspended  after  less 
than  two  weeks  because  of  poor  reli¬ 
ability  of  the  test  missiles.  The  pro¬ 
gram  concentrated  on  an  intensive  re¬ 
liability  improvement  effort.  A  year 
after  the  initial  false  start,  a  full  opera¬ 
tional  test  was  conducted  and  com¬ 
pleted  successfully. 

(3)  The  Maverick  missile  had  a  similar 
experience  of  being  tested  in  an  opera¬ 


tional  environment  before  component 
reliability  testing  was  completed.  As  a 
result,  reliability  failures  had  a  major 
impact  on  the  operational  testers  and 
resulted  in  the  program  being  ex¬ 
tended. 

9.4  ADVANTAGES 
AND  LIMITATIONS 

Before  adopting  a  combined  or  concurrent 
testing  approach,  program  and  test  man¬ 
agers  are  advised  to  consider  the  advan¬ 
tages  and  disadvantages  summarized  in 
Table  9-1. 

9.5  SUMMARY 

A  combined  or  concurrent  testing  approach 
may  offer  an  effective  means  of  shortening 
the  time  required  for  testing  and  achieving 
cost  savings.  If  such  an  approach  is  used, 
extensive  coordination  is  required  to  en¬ 
sure  the  development  and  operational  re¬ 
quirements  are  addressed. 

It  is  possible  to  have  combined  test  teams, 
consisting  of  DT&E  and  OT&E  personnel, 
involved  throughout  the  testing  process. 
The  teams  can  provide  mutual  support  and 
share  mutually  beneficial  data  as  long  as 
the  test  program  is  carefully  planned  and 
executed  and  reporting  activities  are  con¬ 
ducted  separately. 
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10 

PRODUCTION  RELATED 
TESTING  ACTIVITIES 


10.1  INTRODUCTION 

Most  of  the  test  and  evaluation  (T&E)  dis¬ 
cussed  in  this  guidebook  concerns  the  test¬ 
ing  of  the  actual  weapon  or  system  being 
developed,  but  the  program  manager  (PM) 
must  also  evaluate  production-related  test 
activities  and  the  production  process.  This 
chapter  describes  production  management 
and  the  production  process  testing  required 
to  ensure  the  effectiveness  of  the  manufac¬ 
turing  process  and  the  producibility  of  the 
system's  design. 

Normally,  the  development  test  (DT)  and 
operational  test  (OT)  organizations  are  not 
involved  directly  in  this  process.  Usually, 
the  manufacturing  and  quality  assurance 
sections  of  the  program  office  and  a  repre¬ 
sentative  of  the  government  Defense  Plant 
Representatives  Office  (DPRO)  oversee/ 
perform  many  of  these  functions. 

10.2  PRODUCTION  MANAGEMENT 

Production  (manufacturing)  management 
is  defined  as  "the  effective  use  of  resources 
to  produce,  on  schedule,  the  required  num¬ 
ber  of  end  items  that  meet  specified  quality, 
performance,  and  cost.  Production  man¬ 
agement  includes,  but  is  not  limited  to, 
industrial  resource  analysis,  producibility 
assessment,  producibility  engineering  and 
planning,  production  engineering,  indus¬ 
trial  preparedness  planning,  post-produc¬ 
tion  planning,  and  productivity  enhance¬ 
ment,"  (DODD  5000.34).  Production  man¬ 


agement  begins  early  in  the  acquisition 
process,  as  early  as  the  Concept  Explora¬ 
tion  and  Definition  (CED)  Phase,  and  is 
specifically  addressed  at  each  program 
milestone  (MS)  decision  point.  For  instance, 
during  the  CED,  production  feasibility, 
costs  and  risks  should  be  addressed.  Before 
MS  1,  the  PM  must  conduct  an  industrial 
resource  analysis  (IRA)  to  determine  the 
availability  of  production  resources  (e.g., 
capital,  n  iterial,  manpower)  required  to 
support  the  production  of  the  weapon  sys¬ 
tem.  On  the  basis  of  the  results  of  the  IRA, 
critical  materials,  deficiencies  in  the  U.S. 
industrial  base  and  requirements  for  new 
or  updated  manufacturing  technology  can 
be  identified.  Analysis  of  the  industrial- 
base  capacity  is  one  of  the  considerations  in 
preparing  the  Integrated  Program  Sum¬ 
mary  for  the  MS  I  decision.  As  develop¬ 
ment  proceeds,  the  manufacturing  strat¬ 
egy  is  developed;  and  detailed  plans  are 
made  for  the  Production  Phase.  Indepen¬ 
dent  producibility  assessments,  conducted 
in  preparation  for  the  transition  from  de¬ 
velopment  to  production,  are  reviewed  at 
the  MS  II  decision  point.  At  MS  II,  the 
Engineering  and  Manufacturing  Develop¬ 
ment  (EMD)  decision,  the  producibility  of 
the  system  design  concept  is  evaluated  to 
verify  that  the  system  can  be  manufactured 
in  compliance  with  the  production-cost  and 
the  industrial-base  goals  and  thresholds. 

The  MS  III  Production  and  Deployment 
decision  is  supported  by  an  assessment  of 
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the  readiness  of  the  system  to  enter  produc¬ 
tion.  The  system  cannot  enter  the  Full-Rate 
Production  and  Deployment  Phase  until  it 
is  determined  the  principal  contractors  have 
the  necessary  resources  (i.e.,  physical,  fi¬ 
nancial,  and  managerial  capacity)  to  achieve 
the  cost  and  schedule  commitments  and  to 
meet  peacetime  and  mobilization  require¬ 
ments  for  production  of  the  system.  The 
method  of  assessing  production  readiness 
in  preparation  for  MS  III  is  the  Production 
Readiness  Review  (PRR),  which  is  con¬ 
ducted  by  the  PM  and  staff.  An  indepen¬ 
dent  assessment  of  production  readiness  is 
conducted  at  the  same  time  by  the  DOD 
Product  Engineering  Services  Office 
(DPESO)  of  Deputy  Director,  Defense  Re¬ 
search  and  Engineering  (DDDR&E).  The 
DPESO  reports  its  findings  directly  to  the 
defense  acquisition  executive. 

10,3  PRODUCTION  READINESS 
REVIEW  (PRR) 

The  policy,  procedures  and  guidelines  for 
PRRs  are  delineated  in  MIL-STD-1521  as 
follows: 

This  review  is  intended  to  determine 
the  status  of  completion  of  the  specific 
actions  which  must  be  satisfactorily 
accomplished  prior  to  executing  a  pro¬ 
duction  go-ahead  decision.  The  review 
is  accomplished  in  an  incremental  fash¬ 
ion  during  the  Engineering  and  Manu¬ 
facturing  Phase,  usually  two  initial 
reviews  and  one  final  review  to  assess 
the  risk  in  exercising  the  production 
go-ahead  decision.  In  its  earlier  stages 
the  PRR  concerns  itself  with  gross  level 
manufacturing  concerns  such  as  the 
need  for  identifying  high  risk/low 
yield  manufacturing  processes  or  ma¬ 
terials  or  the  requirement  for  manu¬ 
facturing  development  effort  to  sat¬ 
isfy  design  requirements.  Timing  of 
the  incremental  PRR's  is  a  function  of 


program  posture  and  is  not  specifi¬ 
cally  locked  into  other  reviews. 

The  conduct  of  a  PRR  is  the  responsibility 
of  the  PM,  who  usually  appoints  a  director. 
The  director  assembles  a  team  comprised 
of  individuals  in  the  disciplines  of  design, 
industry,  manufacturing,  procurement,  in¬ 
ventory  control,  contracts,  engineering  and 
quality  training.  The  PRR  director  orga¬ 
nizes  and  manages  the  team  effort  and 
supervises  preparation  of  the  findings.  The 
P]^  is  conducted  as  a  time-phased  effort 
during  the  EMD  Phase  following  the  guide¬ 
lines  presented  in  DODI  5000.2. 

10.4  QUALIFICATION  TESTING 

Qualification  testing  is  performed  to  verify 
the  design  and  manufacturing  process,  and 
it  provides  a  baseline  for  subsequent  accep¬ 
tance  tests.  The  production  qualification 
testing  is  conducted  at  the  unit,  subsystem 
and  system  level  on  production  items  and 
is  completed  before  the  production  deci¬ 
sion.  The  results  of  these  tests  are  a  critical 
factor  in  assessing  the  system's  readiness 
for  production.  Downline  production  quali¬ 
fication  tests  are  performed  to  verify  pro¬ 
cess  control  and  may  be  performed  on  se¬ 
lected  parameters  rather  than  at  the  levels 
originally  selected  for  qualification. 

10.4.1  Production  Qualification 
Tests  (PQT) 

Production  qualification  tests  are  a  series  of 
formal  contractual  tests  conducted  to  en¬ 
sure  design  integrity  over  Ine  specified 
operational  and  environmental  range.  The 
tests  are  conducted  on  prototype  or 
preproduction  items  fabricated  to  the  pro¬ 
posed  production  design  drawings  and 
specifications.  The  PQTs  include  all  con¬ 
tractual  reliability  and  maintainability  dem¬ 
onstration  tests  required  prior  to  produc¬ 
tion  release.  For  volume  acquisitions,  these 
tests  are  a  constraint  to  production  release. 


10-2 


10.4.2  First  Article  Tests  (FAT) 

First  article  tests  consist  of  a  series  of  formal 
contractual  tests  conducted  to  ensure  the 
effectiveness  of  the  manufacturing  process, 
equipment  and  procedures.  These  tests  are 
conducted  on  a  random  sample  from  the 
first  production  lot.  These  series  of  tests  are 
repeated  if  the  manufacturing  process, 
equipment  or  procedure  is  changed  signifi¬ 
cantly  and  when  a  second  or  alternative 
source  of  manufacturing  is  brought  on  line. 

10.5  TRANSITION  TO  PRODUCTION 

In  an  acquisition  process,  often  the  first 
indication  that  a  system  will  experience 
problems  is  during  the  transition  from  EMD 
to  low-rate  initial  production.  This  transi¬ 
tion  continues  over  an  extended  period, 
often  months  or  years;  and  during  this 
period,  the  system  is  undergoing  stringent 
contractor  and  government  testing.  There 
may  be  unexpected  failures  requiring  sig¬ 
nificant  design  changes,  which  impact  on 
quality,  producibility,  supportability  and 
may  require  program  schedule  slippage. 

Long  periods  of  transition  usually  indicate 
that  insufficient  attention  to  design  or 
producibility  was  given  early  in  the  Com¬ 
bat  Exploration/Definition  (CED)  or  Dem¬ 
onstration  and  Validation  phases. 

10.5.1  The  Transition  Plan 

Producibility  engineering  and  planning 
(PEP)  is  the  common  thread  that  guides  a 
system  from  CED  to  production.  The  plan 
is  a  management  tool  used  to  ensure  that 
adequate  risk-handling  measures  have  been 
taken  to  transition  from  development  to 
production.  It  contains  a  checklist  to  be 
used  during  the  readiness  reviews.  The 
plan  should  tie  together  the  applications  of 
designing,  testing  and  manufacturing  ac¬ 
tivities  to  reduce  data  requirements,  dupli¬ 


cation  of  effort,  costs  and  scheduling  and  to 
ensure  early  success  of  the  EMD  first  pro¬ 
duction  article. 

10.5.2  Testing  During  the  Transition 

Testing  accomplished  during  the  transi¬ 
tion  from  development  to  pr<^uction  will 
include  acceptance  testing,  manufacturing 
screening  and  final  testing.  These  technical 
tests  are  performed  by  the  contractor  to 
ensure  the  system  will  transition  smoothly 
and  that  test  design  and  manufacturing 
issues  affecting  design  are  addressed.  Dur¬ 
ing  this  same  period,  the  government  will 
be  using  the  latest  available  configuration 
item  to  conduct  the  initial  operational  test 
and  evaluation  (lOT&E).  The  impact  of 
these  tests  may  overwhelm  the  configura¬ 
tion  management  of  the  system  unless  care¬ 
ful  planning  is  accomplished  to  handle  these 
changes. 

10.6  LOW-RATE  INITIAL 
PRODUCTION  (LRIP) 

Low-rate  initial  production  is  the  produc¬ 
tion  of  a  system  in  limited  quantity  to  pro¬ 
vide  articles  for  operational  test  and  evalu¬ 
ation  and  establish  an  initial  production 
base.  Also,  it  permits  an  orderly  increase  in 
the  production  rate  sufficient  to  lead  to  full- 
rate  production  upon  successful  comple¬ 
tion  of  operational  testing.  The  decision  to 
have  an  LRIP  is  made  at  the  MS  II  approval 
of  the  program  acquisition  strategy.  At  that 
time,  the  PM  must  identify:  (1)  the  quantity 
to  be  produced  during  LRIP  and  (2)  the 
quantity  of  LRIP  articles  to  be  used  for 
lOT&E  (to  be  approved  by  the  Director, 
Operational  Test  and  Evaluation  (DOT&E)). 
When  the  decision  authority  thinks  the  sys¬ 
tems  will  not  perform  to  expectation,  he 
will  direct  that  it  not  proceed  beyond  LRIP. 
The  DOT&E  submits  a  report,  on  all  major 
systems,  to  congressional  committees  be¬ 
fore  the  MS  III  decision  to  proceed  beyond 
LRIP  is  made. 


10.7  PRODUCTION  ACCEPTANCE 
TEST  AND  EVALUATION  (PAT&E) 

Production  acceptance  test  and  evaluation 
ensures  that  production  items  demonstrate 
the  fulfillment  of  the  requirements  and 
specifications  of  the  procuring  contract  or 
agreements.  The  testing  also  ensures  the 
system  being  produced  demonstrates  the 
same  performance  as  the  preproduction 
models.  The  procured  items  or  system  must 
operate  in  accordance  with  Type  A  (sys¬ 
tem)  and  Type  C  (production)  specifica¬ 
tions.  The  PAT&E  is  usually  conducted  by 
the  program  office  quality  assurance  sec¬ 
tion  at  the  contractor's  plant  and  may  in¬ 
volve  operational  users. 

For  example,  for  the  Rockwell  B-1 B  Bomber 
production  acceptance,  Rockwell  and  Air 
Force  quality  assurance  inspectors  reviewed 
all  manufacturing  and  ground  testing  re¬ 
sults  for  each  aircraft.  In  addition,  a  flight 
test  team,  composed  of  contractor  and  Air 
Force  test  pilots,  flew  each  aircraft  a  mini¬ 


mum  of  10  hours,  demonstrating  all  on¬ 
board  aircraft  systems  while  in  flight.  Any 
discrepancies  in  flight  were  noted,  corrected 
and  tested  on  the  ground;  they  were  then 
retested  on  subsequent  checkouts  and  ac¬ 
ceptance  flights.  Once  each  aircraft  had 
passed  all  tests  and  all  systems  were  fully 
operational.  Air  Force  authorities  accepted 
the  aircraft.  The  test  documentation  also 
became  part  of  the  delivered  package.  Dur¬ 
ing  this  test  period,  the  program  office 
monitored  each  aircraft's  daily  progress. 

10.8  SUMMARY 

A  primary  purpose  of  production-related 
testing  is  to  lower  the  production  risk  in  a 
major  defense  acquisition  program.  The 
PM  must  ensure  the  contractor's  manufac¬ 
turing  strategy  and  capabilities  will  result 
in  the  desired  product  within  acceptable 
cost.  The  LRIP  and  PAT&E  also  play  major 
roles  in  ensuring  the  production  unit  is 
identical  to  the  design  drawings  and  con¬ 
forms  to  the  specifications  of  the  contract. 
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Table  1 0-1 .  PRR  Guidelines  Checklist 


PRODUCT  DESIGN 

•  Producible  at  low  risk 

•  Stabilized  at  low  rate  of  change 

•  Validated 

•  Reliability,  maintainability  and  performance  demonstrated 

•  Components  engineering  has  approved  all  parts  selections 

INDUSTRIAL  RESOURCES 

•  Adequate  plant  capacity  (peacetime  and  and  wartime  demands) 

•  Facilities,  special  production  and  test  equipment,  and  tooling  identified 

•  Needed  plant  modernization  (CAD/CAM,  other  automation)  accomplished,  which  produces  an  invested  captive 
payback  in  two  to  five  years 

•  Associated  computer  software  developed 

•  Skilled  personnel  and  training  programs  available 

PRODUCTION  ENGINEERING  AND  PLANNING 

•  Production  plan  developed  (reference  MIL-STD-1528) 

•  Production  schedules  compatible  with  delivery  requirements 

•  Manufacturing  methods  and  processes  integrated  with  facilities,  equipment,  tooling  and  plant  layout 

•  Value  engineering  applied 

•  Alternate  production  approaches  available 

•  Drawings,  standards  and  shop  instructions  are  explicit 

•  Configuration  management  adequate 

•  Production  policies  and  procedures  documented 

•  Management  information  system  adequate 

•  Contractor's  management  structure  is  acceptable  to  the  PMO 

•  The  PEP  checklist  has  been  reviewed 

MATERIALS 

•  All  selected  materials  approved  by  contractor’s  materiel  engineers 

•  Bill  of  materials  prepared 

•  “Make-or-Buy’  decisions  complete 

•  Procurement  of  long  lead-time  items  identified 

•  Sole-source  and  government-furnished  items  identified 

•  Contractor’s  inventory-control  system  adequate 

•  Contractor’s  material  cost  procurement  plan  complete 

QUALITY  ASSURANCE  (QA) 

•  Quality  plan  in  accordance  with  contract  requirements 

•  Quality  control  procedures  and  acceptance  criteria  established 

•  QA  organization  participates  in  production  planning  effort 

LOGISTICS 

•Operational  support,  test  and  diagnostic  equipment  available  at  system  deployment 
•Training  aids,  simulators  and  other  devices  ready  at  system  deployment 
•Spares  integrated  into  production  lot  flow 


10-5 


Ill 

MODULE 

Operational 
Test  and  Evaluation 


Operational  Test  and  Evaluation  (OT&E)  is 
conducted  to  ensure  a  weapon  system  meets 
the  validated  requirements  of  the  user  in  a 
realistic  scenario.  Operational  tests  are  fo¬ 
cused  on  operational  requirements,  effec¬ 
tiveness  and  suitability,  and  not  on  the 
proof  of  engineering  specifications,  as  is 
the  case  with  development  testing.  This 
module  provides  an  overview  of  OT&E 
and  discusses  how  OT&E  results  provide 
essential  information  for  milestone  deci¬ 
sions. 


11 

INTRODUCTION  TO  OPERATIONAL 
TEST  AND  EVALUATION 


11.1  INTRODUCTION 

This  chapter  provides  an  introduction  to 
the  concept  of  operational  test  and  evalua¬ 
tion  (OT&E).  It  outlines  the  purpose  of 
OT&E,  discusses  the  primary  participants 
in  theOT&Eprocess,describes several  types 
of  OT&E,  and  includes  some  general  guide¬ 
lines  for  the  successful  planning,  execution 
and  reporting  of  OT&E  programs. 

11.2  PURPOSE  OF  OT&E 

Operational  test  and  evaluation  is  con¬ 
ducted  for  major  programs  by  an  organiza¬ 
tion  that  is  independent  of  the  developing, 
procuring  and  using  commands.  It  is  nor¬ 
mally  conducted  in  phases,  each  of  which 
are  keyed  to  a  decision  review  in  the  mate¬ 
riel  acquisition  process.  It  is  conducted  with 
typical  user  operators,  crews  or  units  in 
realistic  and  operational  environments.  The 
OT &E  provides  the  decision  authority  with 
an  estimate  of: 

(1)  The  degree  of  satisfaction  of  the  user's 
requirements  expressed  as  operational  ef¬ 
fectiveness  and  suitability  of  the  new  sys¬ 
tem; 

(2)  The  system's  desirability,  consider¬ 
ing  equipment  already  available,  and  op¬ 
erational  benefits  or  burdens  associated 
with  the  new  system; 

(3)  The  need  for  further  development  of 
the  new  system; 


(4)  The  adequacy  of  doctrine,  organiza¬ 
tions,  operating  techniques,  tactics  and 
training  for  employment  of  the  system;  of 
maintenance  support  for  the  system;  and  of 
the  system's  performance  in  the  coimter- 
measures  environment. 

11.3  TEST  PARTICIPANTS 

The  OT&E  of  developing  systems  is  man¬ 
aged  by  an  independent  testing  agency, 
which  each  Service  is  required  to  maintain. 
It  is  accomplished  under  conditions  of  op¬ 
erational  realism  whenever  possible.  Per¬ 
sonnel  who  operate,  maintain  and  support 
the  system  during  OT&E  are  trained  to  a 
level  commensurate  with  that  of  personnel 
who  will  perform  these  functions  under 
peacetime  and  wartime  conditions.  Also, 
program  management  office  (PMO)  per¬ 
sonnel  and  test  coordinating  groups  play 
important  parts  in  the  overall  OT&E  plan¬ 
ning  and  execution  process. 

11.3.1  Program  Management  Office 

Even  though  operational  testing  is  per¬ 
formed  by  an  independent  organization, 
the  program  manager  (PM)  plays  an  im¬ 
portant  role  in  its  planning,  reporting  and 
funding.  He  must  coordinate  program  ac¬ 
tivities  with  the  test  community,  espe¬ 
cially  the  operational  test  agencies.  He  en¬ 
sures  that  testing  can  address  the  critical 
issues,  and  provides  feedback  from  OT&E 
testing  activities  to  contractors. 
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At  each  milestone  review,  the  PM  is  re¬ 
quired  to  brief  the  decision  authority  on  the 
testing  planned  and  completed  on  the  pro¬ 
gram.  It  is,  therefore,  important  that  PMO 
personnel  have  a  good  understanding  of 
the  test  program  and  that  they  work  with 
the  operational  test  community.  This  will 
ensure  OT&E  is  well-planned  and  adequate 
resources  are  available.  The  PMO  should 
involve  the  test  community  by  organizing 
test  coordinating  groups  at  program  initia¬ 
tion  and  by  establishing  channels  of  com¬ 
munication  between  the  PMO  and  the  key 
testorganizations.  The  PMOcan  often  avoid 
misunderstandings  by  aggressively  moni¬ 
toring  the  system  testing  and  providing 
up-to-date  information  to  key  personnel  in 
OSD  and  the  Services.  The  PMO  staff  should 
keep  appropriate  members  of  the  test  com¬ 
munity  well-informed  concerning  system 
problems  and  the  actions  taken  by  the  PMO 
to  correct  them. 

11.3.2  Test  Coordinating  Groups 

The  test  and  evaluation  (T&E)  working 
groups,  such  as  the  Army's  Test  Integra¬ 
tion  Working  Groups  (TIWG)  and  Air 
Force's  Test  Planning  Working  Groups 
(TPWG),  are  chartered  by  their  respective 
PMO  to  coordinate  and  integrate  the  plan¬ 
ning  and  execution  of  the  T&E  program. 
The  Army  and  Air  Force  groups  are  chaired 
by  a  representative  of  the  PMO,  often  the 
deputy  for  test  and  evaluation  or  systems 
engineer.  Members  of  these  groups  repre¬ 
sent  various  communities  including  the 
user,  development  and  operational  testing, 
independent  evaluation,  logistics,  training 
and  contractor,  as  appropriate.  The  func¬ 
tions  of  the  groups  are  to:  facilitate  the  use 
of  testing  expertise,  instrumentation,  facili¬ 
ties,  simulations  and  models;  integrate  test 
requirements;  accelerate  the  TEMP  coordi¬ 
nation  process;  resolve  test  cost  and  sched¬ 
uling  problems;  and  provide  a  forum  to 
ensure  T&E  of  the  system  is  coordinated. 
The  existence  of  a  test  coordinating  group 
does  not  alter  the  responsibilities  of  any 


command  or  headquarters;  and,  in  the  event 
of  disagreement  within  a  group,  the  issue  is 
resolved  through  the  normal  command/ 
staff  channels.  Within  the  Air  Force,  the 
TPWG  may  help  to  prepare  the  test  por¬ 
tions  of  the  request  for  proposal  and  re¬ 
lated  contractual  documentation  and  to 
evaluate  the  contractors'  proposals.  In  all 
Services,  the  groups  help  develop  the 
TEMP. 

11.3.3  Service  Operational  Test  Agencies 

The  operational  test  and  evaluation  agen¬ 
cies  (OTA)  should  become  involved  early 
in  the  system's  life  cycle,  usually  before 
program  starts  at  Milestone  (MS)  I.  At  this 
time,  they  can  begin  to  develop  strategies 
for  conducting  of  operational  tests.  As  test 
planning  continues,  a  more-detailed  Test 
and  Evaluation  Master  Plan  (TEMP)  is  de¬ 
veloped  and  the  test  resources  are  identi¬ 
fied  and  scheduled.  During  the  early  stages, 
the  OTAs  structure  an  OT&E  program  con¬ 
sistent  with  the  approved  acquisition  strat¬ 
egy  for  the  system,  identify  critical  opera¬ 
tional  test  issues,  and  assess  the  adequacy 
of  candidate  systems.  As  the  program 
moves  into  advanced  planning,  OT&E  ef¬ 
forts  are  directed  toward  becoming  famil¬ 
iar  with  the  system,  encouraging  interface 
between  the  user  and  developer  and  fur¬ 
ther  refining  the  critical  operational  issues. 
Each  Service  has  an  independent  organiza¬ 
tion  dedicated  to  planning,  executing  and 
reporting  the  results  of  that  Service's  OT&E 
activities.  These  organizations  are  the:  Army 
Operational  Test  and  Evaluation  Command 
(OPTEC),  Navy  Operational  Test  and 
Evaluation  Force  (OPTEVFOR),  Air  Force 
Operational  Test  and  Evaluation  Center 
(AFOTEC),  and  Marine  Corps  Operational 
Test  and  Evaluation  Activity  (MCOTEA). 

11.3.4  Test  Personnel 

Operational  testing  is  conducted  on  mate¬ 
riel  systems  with  "typical"  user  players  in 
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a  realistic  operational  environment.  It  uses 
persormel  with  the  same  military  occupa¬ 
tional  specialties  as  those  who  will  operate, 
maintain  and  support  the  system  when 
fielded.  Participants  are  trained  in  the 
system's  operation  based  on  the  Service's 
operational  mission  profiles.  Because  some 
operational  tests  consist  of  force-on-force 
tests,  the  forces  opposing  the  tested  system 
must  also  be  trained  in  threat  tactics  and 
doctrine.  For  operational  testing  conducted 
before  initial  operational  test  and  evalua¬ 
tion  (lOT&E),  most  system  training  is  con¬ 
ducted  by  the  system's  contractor.  For 
lOT&E,  the  contractor  trains  the  school 
cadre  who  train  other  troops.  As  the  system 
enters  full-rate  production,  the  Services 
assume  training  responsibilities. 

11.4  TYPES  OF  OT&E 

Operational  Test  and  Evaluation  can  be 
subdivided  into  two  phases:  operational 
testing  performed  before  MS  III  (full-rate 
production /deployment  decision)  and  the 
operational  testing  performed  after  MS  III. 
The  Pre-MS  III  OT&E  includes  operational 
assessments  and  lOT&E.  Operational  as¬ 
sessments  begin  early  in  the  program,  fre¬ 
quently  before  program  start  (MS  I)  and 
continue  until  the  system  is  certified  as 
ready  for  lOT&E.  The  initial  operational 
test  and  evaluation  is  conducted  just  before 
the  full-rate  production/deployment  deci¬ 
sion.  After  full-rate  production /deploy¬ 
ment,  all  subsequent  operati'^nal  testing  is 
referred  to  as  follow-on  operational  test 
and  evaluation  (FOT&E).  In  the  Air  Force, 
if  no  research  and  development  funding  is 
committed  to  a  system,  qualification  OT&E 
may  be  performed  in  lieu  of  lOT&E.  The 
Navy  uses  the  term  "OPEVAL"  to  define 
lOT&E. 

11.4.1  Early  Operational  Assessments 

Early  operational  assessments  are  con¬ 
ducted  primarily  to  forecast  and  evaluate 


the  potential  operational  effectiveness  and 
suitability  of  the  weapon  system  during 
development.  Early  operational  assess¬ 
ments  start  in  the  Concept/  Definition  Phase 
and  are  conducted  on  the  developing  sys¬ 
tem  until  MS  II. 

11.4.1.1  Operational  Assessments 

Operational  assessments  begin  after  MS  II, 
when  the  OTAs  start  their  evaluations  of 
system-level  performance.  The  OTA  uses 
any  testing  results  and  data  from  other 
sources  during  an  assessment.  These  data 
are  evaluated  by  the  OTA  from  an  opera¬ 
tional  point  of  view.  As  the  program  ma¬ 
tures,  these  operational  assessment  require¬ 
ments  are  conducted  on  preproduction  ar¬ 
ticles  until  the  system  is  fully  developed 
and  certified  ready  for  its  lOT&E  or 
OPEVAL  in  the  Navy. 

11.4.1.2  Initial  Operational 

Test  and  Evaluation  (Navy  OPEVAL) 

Initial  oper  ational  test  and  evaluation  is  the 
final  dedicated  phase  of  OT&E  preceding  a 
full-rate  production  decision.  It  is  the  final 
evaluation  that  entails  dedicated  opera¬ 
tional  testing  of  production-representative 
test  articles  and  uses  typical  operational 
personnel  in  a  scenario  that  is  as  realistic  as 
possible.  The  lOT&E  is  conducted  by  an 
OT&E  agency  independent  of  the  contrac¬ 
tor,  PMO  or  developing  agency.  The  test  is 
defined  in  DODI  5000.2  as: 

All  operational  test  and  evaluation  con¬ 
ducted  on  production  or  production 
representative  articles,  to  support  the 
decision  to  proceed  beyond  low-rate 
initial  production.  It  is  conducted  to 
provide  a  valid  estimate  of  expected 
system  operational  effectiveness  and 
operational  suitability.  The  definition 
of  "OT&E"  as  spelled  out  in  congres¬ 
sional  legislation  (see  glossary)  is  gen- 
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erally  conFidered  applicable  only  to 
initial  operational  test  and  evaluation 
(lOT&E). 

Further,  lOT&E  must  be  conducted 
without  system  contractor  personnel 
participation  in  any  capacity  other  than 
stipulated  in  service  wartime  tactics 
and  doctrine  as  set  forth  in  Public  Law 
99-661  by  Congress.  The  results  from 
this  test  are  evaluated  and  presented 
to  the  milestone  decision  authority  (i.e. 
MS  III,  the  decision  to  enter  full-rate 
production)  to  support  the  beyond- 
LRIP  decision.  This  phase  oi  OT&E 
addresses  the  critical  issues  identified 
in  the  Operational  Requirements  Docu¬ 
ment  (ORD)  and  the  TEMP. 

11.4.2  Follow-On  Operat'onal 
Test  and  Evaluation 

Follow-on  operational  test  and  evaluation 
is  conducted  after  the  MS  III  decision.  The 
tests  are  conducted  in  a  realistic  tactical 
environment  similar  to  that  used  in  lOT &E, 
but  many  test  items  may  be  used.  Normal¬ 
ly  FOT&E  is  conducted  using  fielded  pro¬ 
duction  systems.  Specific  objectives  of 
FOT&E  include  testing  modifications  that 
are  to  be  incorporated  into  production  sys¬ 
tems,  completing  any  deferred  or  incom¬ 
plete  lOT&E,  and  assessing  reliability  in¬ 
cluding  spares  support.  The  tests  are  also 
used  to  evaluate  the  system  in  a  different 
platform  application  for  new  tactical  appli¬ 
cations  or  against  new  threats. 

11.4.3  Qualification  Operational 
Test  and  Evaluation 

Air  Force  qualification  operational  test  and 
evaluation  maybe  performed  by  the  major 
command,  user  or  AFOTEC  and  is  con¬ 
ducted  on  minor  modifications  or  new  ap¬ 
plications  of  existing  equipment  when  no 
research  and  development  funding  is  re¬ 


quired.  An  example  of  a  program  in  which 
(^T&E  was  performed  by  the  Air  Force  is 
the  A-10  Air-to-Air  Self  Ciefense  Program. 
In  this  program  the  mission  of  the  A-10  was 
expanded  from  strictly  ground  support  to 
include  an  air-to-air  defense  role.  To  ac¬ 
complish  this  the  A-10  aircraft  was  modi¬ 
fied  with  off-the-shelf  AIM-9  and  air-to-air 
missiles;  QOT&E  was  performed  on  the 
system  to  evaluate  its  operational  effective¬ 
ness  and  suitability. 

11.5  TEST  PLANNING 

Operational  test  planning  is  one  of  the  most 
important  parts  of  the  OT&E  process. 
Proper  planning  facilitates  the  acquisition 
of  data  to  support  the  determination  of  the 
weapon  system's  operational  effectiveness 
and  suitability.  Planning  must  be  pursued 
in  a  deliberate,  comprehensive  and  struc¬ 
tured  manner.  Careful  and  complete  plan¬ 
ning  may  not  guarantee  a  successful  test 
program;  but  inadequate  planning  can  re¬ 
sult  in  significant  test  problems,  poor  sys¬ 
tem  performance  and  cost  overruns.  Op¬ 
erational  test  planning  is  conducted  by  the 
OTA  before  program  start,  and  more-de¬ 
tailed  planning  usually  starts  about  two 
years  before  each  operational  test  event. 

Operational  planning  can  be  divided  into 
three  phases:  early  planning,  advanced 
planning  and  detailed  planning.  Early  plan¬ 
ning  entails  developing  critical  operational 
issues,  formulating  a  plan  for  evaluations, 
determining  the  concept  of  operation,  en¬ 
visioning  the  operational  environment  and 
developing  mission  scenarios  and  resource 
requirements.  Advanced  planning  encom¬ 
passes  the  determination  of  the  purpose 
and  scope  of  testing  and  identification  of 
measures  of  effectiveness  (MOEs)  for  criti¬ 
cal  issues.  It  includes  developing  test  objec¬ 
tives,  establishing  a  test  approach,  and  esti¬ 
mating  test  resource  requirements.  Detailed 
planning  involves  developing  step-by-step 
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procedures  to  be  followed  as  well  as  the 
fa  i«i  coordination  of  resource  requirements 
necessary  to  carry  out  OT&E. 

11.5.1  Testing  Critical  Operational 
Issues  (COD 

Part  15  of  DODI  5000.2  defines  a  critical 
operational  issue  as: 

A  key  operational  effectiveness  or  op¬ 
erational  suitability  issue  that  must  be 
examined  in  operational  test  and  evalu¬ 
ation  to  determine  the  system's  capa¬ 
bility  to  perform  its  mission.  A  critical 
operational  issue  is  normally  phrased 
as  a  question  to  be  answered  in  evalu¬ 
ating  a  system's  operational  effective¬ 
ness  and/or  operational  suitability 

One  of  the  purposes  of  OT&E  is  to  resolve 
COIs  about  the  system.  The  first  step  in  an 
OT&E  program  is  to  identify  these  critical 
issues,  some  of  which  are  explicit  in  the 
ORD.  Examples  can  b  e  found  in  questions 
such  as:  "How  well  does  the  system  per¬ 
form  a  particular  aspect  of  its  mission?" 
"Can  the  system  be  supported  ’.ogistically 
in  the  field?"  Other  issues  arise  from  ques¬ 
tions  asked  about  system  performance  or 
how  it  will  affect  other  systems  with  which 
it  must  operate.  Critical  issues  provide  fo¬ 
cus  and  direction  for  the  operational  test. 
Identifying  the  issues  is  analogous  to  the 
first  step  in  the  system  engineering  pro¬ 
cess,  that  is,  defining  the  problem.  When 
critical  operational  issues  are  properly  ad¬ 
dressed,  deficiencies  in  the  system  can  be 
uncovered.  They  form  the  basis  for  a  struc¬ 
tured  technique  of  analysis  by  which  de¬ 
tailed  subobjectives  or  MOEs  can  be  estab¬ 
lished.  During  the  operational  test,  each 
subobjective  is  addressed  by  an  actual  test 
measurement.  After  these  issues  are  identi¬ 
fied,  the  evaluation  plans  and  test  design 
are  developed  for  test  execution.  (For  more 
information,  see  the  chapter  on  evalua¬ 
tion.) 


11.5.2  Test  Ivealism 

Test  realism  for  OT&E  will  vary  directly 
with  the  degree  of  system  maturity.  Efforts 
early  in  the  acquisition  program  should 
focus  on  active  involvement  of  users  and 
operationally  oriented  environments.  Fi¬ 
delity  of  the  "combat  environment"  should 
peak  during  the  lOT&E  when  force-on- 
force  testing  of  the  production  representa¬ 
tive  system  is  conducted.  The  degree  of 
success  in  replicating  a  realistic  operational 
environment  has  a  direct  impact  on  the 
credibility  of  the  lOT&E  test  report.  Areas 
of  primary  concern  for  the  test  planner  can 
be  derived  from  the  legislated  definition  of 
OT&E: 

(1)  A  field  test  includes  ali  of  the 
elements  normally  expected  to  be  en¬ 
countered  in  the  operational  arena, 
such  as  appropriate  size  and  type  of 
maneuver  terrain,  environmental  fac¬ 
tors,  day/ night  operations,  austere  liv¬ 
ing  conditions,  etc. 

(2)  Realistic  combat  should  be  repli¬ 
cated  using  appropriate  tactics  and 
doctrine,  representative  threat  forces 
properly  trained  in  the  employment  of 
threat  equipment,  free  play  responses 
to  test  stimulus,  stress,  "dirty"  battle 
area  (fire,  smoke,  NBC,  ECM,  etc.), 
wartime  tempo  to  operations,  real  time 
casualty  assessment,  and  forces  requir¬ 
ing  interoperability. 

(3)  Any  item  means  the  production 
representative  configuration  of  the 
system  at  that  point  in  time,  including 
appropriate  logistics  tail. 

(4)  Typical  military  users  are  obtained 
by  taking  a  cross  section  of  adequately 
trained  skill  levels  and  ranks  of  the 
intended  operational  force.  Selection 
of  "golden  crews"  or  the  best  of  the 
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best  does  not  provide  test  data  reflect¬ 
ing  the  successes  nor  problems  of  the 
"murphy  and  gang"  of  typical  units. 

In  his  book.  Operational  Test  and  Evaluation, 
Roger  Stevens  states,  "In  order  to  achieve 
realism  effectively  in  an  OT&E  program,  a 
concern  for  realism  must  pervade  the  en¬ 
tire  test  program  from  the  very  beginning 
of  test  planning  to  the  time  when  the  very 
last  test  iteration  is  run."  Realism  is  a  sig¬ 
nificant  issue  during  planning  and  execu¬ 
tion  of  OT&E. 

11,5.3  Selection  of  a  Test  Concept 

An  important  step  in  the  development  of 
an  OT&E  program  is  to  develop  an  overall 
test  program  concept.  Determinations  must 
be  made  regarding  when  OT&E  will  be 
conducted  during  systems  development, 
what  testing  is  to  be  done  on  production 
equipment,  how  the  testing  will  be  evolu¬ 
tionary,  and  what  testing  will  have  to  wait 
until  all  system  capabilities  are  developed. 
This  concept  can  best  be  developed  by  con¬ 
sidering  a  number  of  aspects  such  as  test 
information  requirements,  system  availabil¬ 
ity  for  test  periods,  and  the  demonstration 
of  system  capabilities.  The  test  concept  is 
driven  by  the  acquisition  strategy  and  is  a 
road  map  used  for  planning  test  and  evalu¬ 
ation  events. 

11.6  TEST  EXECUTION 

An  operational  test  plan  is  only  as  good  as 
the  execution  of  that  plan.  The  execution  is 
the  essential  bridge  between  test  planning 
and  test  reporting.  The  test  is  executed 
through  the  OTA  test  director's  efforts  and 
the  actions  of  the  test  team.  For  successful 
execution  of  the  OT&E  plan,  the  test  direc¬ 
tor  must  direct  and  control  the  test  re¬ 
sources  and  collect  the  data  required  for 
presentation  to  the  decision  authority.  He 
must  prepare  for  testing,  activate  and  train 


the  test  team,  develop  test  procedures  and 
operating  instructions,  control  data  man¬ 
agement,  create  OT&E  plan  revisions,  and 
manage  each  of  the  test  trials.  His  data 
managtiiicnt  duties  will  encompass  col¬ 
lecting  raw  data,  creating  a  data  status 
matrix,  ensuring  data  quality,  processing 
.^nd  reducing,  verifying,  filing,  storing,  re¬ 
trieving  and  analyzing.  Once  all  tests  have 
been  completed  and  the  data  is  reduced 
and  analyzed,  the  results  must  be  reported. 
A  sample  test  organization  used  for  the 
Army  OT&E  of  the  improved  81mm  mor¬ 
tar  is  illustrated  in  Figure  1 1-1 .  (In  the  Army, 
the  Deputy  Test  Director  comes  from  the 
OTA  and  controls  the  daily  operational  test 
activity.) 

11.7  TEST  REPORTING 

The  lOT&E  test  report  is  a  very  important 
document.  It  must  communicate  the  re¬ 
sults  of  completed  tests  to  decision  authori¬ 
ties  in  a  timely,  factual,  concise,  compre¬ 
hensive  and  accurate  manner.  The  report 
must  present  a  balanced  view  of  the  weapon 
system's  successes  and  problems  during 
testing,  illuminating  both  the  positive  as¬ 
pects  and  system  deficiencies  discovered. 
Analysis  of  test  data  and  their  evaluation 
may  be  in  one  report  (USAF,  USN)  or  in 
separate  documents  (USA,  USMC). 

There  are  four  types  of  reports  most  fre¬ 
quently  used  in  reporting  OT&E  results. 
These  include  status,  interim,  quick-look 
and  final  reports.  The  status  report  gives 
periodic  updates  (e.g.,  monthly,  quarterly) 
and  reports  recent  test  findings  (discreet 
events  such  as  missile  firings).  The  interim 
report  provides  a  summary  of  the  cumula¬ 
tive  test  results  to  date  when  there  is  an 
extended  period  of  testing.  The  quick-look 
reports  provide  preliminary  test  results, 
are  usually  prepared  immediately  after  a 
test  event  (less  than  7  days)  and  have  been 
used  to  support  program  decision  mile- 
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TES 


Figure  11-1.  Organizational  Breakdown  of  the  1-81  mm  Mortar  Operational  Test  Directorate 


stones.  The  final  T&E  report  (Air  Force, 
Navy)  or  independent  evaluation  report 
(Army,  Marine)  presents  the  conclusions 
and  recommendations  including  all  sup¬ 
porting  data  and  covering  the  entire  lOT&E 
program. 

11.8  SUMMARY 

The  purpose  of  OT&E  is  to  assess  opera¬ 
tional  effectiveness  and  suitability  at  each 
stage  in  the  acquisition  process.  Opera¬ 
tional  effectiveness  is  a  measure  of  the  con¬ 
tribution  of  the  system  to  mission  accom¬ 
plishment  under  actual  conditions  of  em¬ 
ployment.  Operational  suitability  is  a  mea¬ 


sure  of  the  maintainability  and  reliability 
of  the  system;  the  effort  and  level  of  train¬ 
ing  required  to  maintain,  support  and  op¬ 
erate  it;  and  any  unique  logistic  or  training 
requirements  it  may  have.  The  OT&E  may 
provide  information  on  tactics,  doctrine, 
organization  and  personnel  requirements 
and  may  be  used  to  assist  in  the  prepara¬ 
tion  of  operating  and  maintenance  instruc¬ 
tions  and  other  publications.  One  of  the 
most  important  aspects  is  that  OT&E  pro¬ 
vides  an  independent  evaluation  of  the 
degree  of  progress  made  toward  satisfying 
the  user's  requirements  during  the  system 
development  process. 
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12 

OT&E  TO  SUPPORT 
MILESTONE  DECISIONS 


12.1  INTRODUCTION 

Mindful  of  principles  of  objectivity  and 
impartial  evaluation,  operational  test  and 
evaluation  (OT&E)  is  conducted  before  each 
major  milestone  review  to  provide 
the  decision  authori.^  with  the  latest  re¬ 
sults  from  testing  of  critical  operational 
issues.  The  philosophy  of  OT&E  has  been 
related  to  three  terms  —  adequacy,  quality 
and  credibility: 

Adequacy  —  The  amount  of  data  and 
realism  of  test  conditions  must  be  sufficient 
to  support  the  evaluation  of  the  critical 
operational  issues. 

Quality  —  The  test  planning,  control  of 
test  events,  and  treatment  of  data  must 
provide  clear  and  accurate  test  reports. 

Credibility — The  conduct  of  the  test  and 
data  handling  must  be  separated  from  ex¬ 
ternal  influence  and  personal  biases. 

Operational  testing  is  conducted  to  pro¬ 
vide  information  to  support  DOD  execu¬ 
tive-level  management  decisions  on  major 
acquisition  programs.  Operational  test  and 
evaluation  is  accomplished  using  a  test 
cycle  of  successive  actions  and  documents. 
During  the  early  stages  of  the  program,  the 
process  is  informal  and  modified  as  neces¬ 
sary.  As  programs  mature,  documentation 
for  major  systems  and  those  designated  by 
the  Director,  Operational  Test  and  Evalua¬ 
tion  (DOT&E)  for  oversight  must  be  sent  to 
the  Office  of  the  Secretary  of  Defense  (OSD) 


for  approval  before  the  testing  can  be  con¬ 
ducted  or  the  systems  can  be  cleared  to 
proceed  into  low-rate  initial  production 
(LRIP).  Figure  12-1  illustrates  how  OT&E 
relates  to  the  acquisition  process. 

12.2  OT&E  DURING  THE  CONCEPT 
EXPLORATION/DEFINITION  PHASE 
(MS  0  to  MS  I) 

The  OT&E  conducted  during  the  Concept 
Exploration  and  Definition  (CED)  Phase  is 
an  early  operational  assessment  (OA)  fo¬ 
cused  on  investigating  the  deficiencies  iden¬ 
tified  during  the  mission  area  analysis. 
Operational  testers  participate  in  these 
evaluations  to  validate  the  OT&E  require¬ 
ments  for  future  testing  and  to  identify 
issues  and  criteria  that  can  only  be  resolved 
through  OT &E  to  initiate  early  test  resource 
planning. 

Before  MS  I,  the  OT&E  objectives  are  to 
assist  in  evaluating  alternative  concepts  to 
solve  the  mission  area  deficiencies  and  to 
assess  the  operational  impact  of  the  sys¬ 
tem.  This  early  assessment  also  provides 
data  to  support  a  decision  on  whether  to 
enter  the  Demonstration  and  Validation 
Phase.  The  OT&E  conducted  during  the 
CED  Phase  supports  developing  estimates 
of: 

(1)  The  military  need  for  the  proposed 
system; 
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(2)  A  demonstration  that  there  is  a  sound 
physical  basis  for  a  new  system; 

(3)  An  analysis  of  concepts,  based  on 
demonstrated  physical  phenomena,  for  sat¬ 
isfying  the  military  need; 

(4)  The  system's  affordability  and  life- 
cycle  cost; 

(5)  The  ability  of  a  modification  to  an 
existing  U.S.  or  allied  system  to  provide 
needed  capability; 

(6)  An  operational  utility  assessment; 

(7)  An  impact  of  the  system  on  the  force 
structure. 

At  MS  I,  there  is  normally  no  hardware 
available  for  the  operational  tester.  There¬ 
fore,  the  early  operational  assessment  is 
conducted  from  surrogate  test  and  experi¬ 
ment  data,  breadboard  models,  factory  user 
trials,  mock-up /simulators,  and  user  dem¬ 
onstrations  (Figure  12-2).  This  makes  early 
assessments  difficult,  and  some  areas  can¬ 
not  be  covered  in-depth.  However,  these 
assessments  provide  vital  introductory  in¬ 
formation  on  the  system's  potential  opera¬ 
tional  utility. 

The  OT&E  products  from  this  pnase  of 
testing  include  the  information  provided 
to  the  decision  authority,  data  collected  for 
further  evaluation,  input  to  the  Test  and 
Evaluation  Master  Plan  (TEMP)  and  early 
test  and  evaluation  (T&E)  planning.  Spe¬ 
cial  logistics  problems,  program  objectives, 
program  plans,  performance  parameters 
and  acquisition  strategy  are  pri¬ 

mary  influence  to  the  operational  tester 
during  this  phase  and  must  be  carefully 
evaluated  to  project  the  system's  opera¬ 
tional  effectiveness  and  suitability. 


12.3  OT&E  DURING  THE 
DEMONSTRATION  AND 
VALIDATION  PHASE 
(MS  I  to  MS  II) 

Combined  development  test  (DT)/OT&E 
or  an  early  operational  assessment  during 
the  Demonstration  and  Validation  Phase, 
is  conducted  to  support  the  MS  II  decision 
regarding  a  system's  readiness  to  move 
into  the  Engineering  and  Manufacturing 
(EMD)  Phase.  In  all  cases,  appropriate  T&E 
must  be  conducted  before  the  MS  II  deci¬ 
sion,  thereby  providing  data  for  identifica¬ 
tion  of  risk  before  more  resources  are  com¬ 
mitted.  As  appropriate,  LRIP  may  be  ap¬ 
proved  at  MS  II  to  verify  production  capa¬ 
bility  and  to  provide  test  resources  needed 
to  conduct  interoperability,  live  fire,  or  op¬ 
erational  testing. 

12.3.1  Objectives  of  Early 
Operational  Assessments 

Early  operational  assessments  are  con¬ 
ducted  to  facilitate  identification  of  the  best 
design,  indicate  the  risk  level  of  perfor¬ 
mance  for  this  phase  of  the  development, 
examine  operational  aspects  of  the  system's 
development,  and  estimate  potential  op¬ 
erational  effectiveness  and  suitability.  Ad¬ 
ditionally,  an  analysis  of  the  planning  for 
transition  from  development  to  produc¬ 
tion  is  initiated.  Early  operational  assess¬ 
ments  supporting  the  MS  II  decision  are 
intended  to: 

(1)  Assess  the  potential  of  the  new  sys¬ 
tem  in  relation  to  existing  capabilities; 

(2)  Assess  system  effectiveness  and  suit¬ 
ability  so  that  affordability  can  be  evalu¬ 
ated  for  program  cost  vs.  military  utility; 

(3)  Assess  the  adequacy  of  the  concept 
for  employment,  supportability  and  orga- 


12-3 


12-4 


nization;  doctrinal,  tactical  and  training 
requirements;  and  related  critical  issues; 

(4)  Estimate  the  need  for  the  selected 
systems  in  consideration  of  the  threat  and 
system  alternatives  based  on  military  util¬ 
ity; 

(5)  Assess  the  validity  of  the  operational 
concept; 

(6)  List  the  key  risk  areas  and  critical 
operational  issues  that  need  to  be  resolved 
before  EMD  is  initiated; 

(7)  Assess  the  need  for  LRIP  of  hard¬ 
ware  to  support  initial  operational  test  and 
evaluation  (lOT&E)  prior  to  the  full-rate 
production  decision; 

(8)  Provide  data  to  support  test  planning 
for  the  EMD  Phase. 

During  this  phase,  OT &E  may  be  conducted 
on  brassboard  configurations,  experimen¬ 
tal  prototypes  or  advanced  development 
prototypes.  Dedicated  test  time  may  be 
made  available  for  the  operational  tester. 
However,  the  OT&E  assessments  may  also 
make  use  of  many  other  additional  data 
sources.  Examples  of  additional  sources 
often  used  by  the  Army  during  this  phase 
include:  concept  evaluation  program  tests, 
innovative  testing,  force  development  tests 
and  experimentation  (FDT&E),  source  se¬ 
lection  tests,  user  participation  in  develop¬ 
ment  test  and  evaluation  (DT&E)  and  op¬ 
erational  feasibility  tests.  The  results  from 
this  testing,  analysis  and  evaluation  are 
documented  in  the  Integrated  Program 
Summary  (IPS).  These  data,  along  with  the 
mission  needs  and  requirements  documen¬ 
tation  and  TEMP,  assist  in  the  review  of 
performance  for  the  MS  II  decision. 


12.4  OT&E  DURING  THE 
ENGINEERING  AND 
MANUFACTURING 
DEVELOPMENT  PHASE 
(MS  II  to  MS  III) 

The  lOT&E  and  operational  assessments 
during  the  EMD  Phase  are  conducted  on 
engineering  development  models  or  pro¬ 
duction  representative  systems.  These  op¬ 
erational  evaluations  estimate  the  opera¬ 
tional  effectiveness  and  suitability  and  pro¬ 
vide  data  on  whether  the  system  meets 
minimum  operational  thresholds.  Just  be¬ 
fore  the  full-rate  production  MS  III  deci¬ 
sion,  the  dedicated  T&E  is  conducted  on 
equipment  that  has  been  formally  certified 
by  the  program  manager  as  being  ready  for 
the  "final  OT&E."  This  dedicated  lOT&E  is 
conducted  in  a  test  environment  as  opera¬ 
tionally  realistic  as  possible. 

12.4.1  OT&E  Objectives 

The  OA/IOT&E  conducted  during  EMD, 
is  characterized  by  testing  performed  by 
user  organizations  in  a  field  exercise  to 
examine  the  organization  and  doctrine,  in¬ 
tegrated  logistics  support,  threat,  commu¬ 
nications,  command  and  control,  and  tac¬ 
tics  associated  with  the  operational  em¬ 
ployment  of  the  unit  during  tactical  opera¬ 
tions.  This  includes  estimates  which: 

(1)  Assess  operational  effectiveness  and 
suitability; 

(2)  Assess  the  survivability  of  the  sys¬ 
tem; 

(3)  Assess  the  systems  reliability,  main¬ 
tainability  and  plans  for  integrated  logis¬ 
tics  support; 

(4)  Evaluate  manpower,  personnel,  train¬ 
ing  and  safety  requirements; 
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(5)  V alidate  organizational  and  employ¬ 
ment  concepts; 

(6)  Determine  training  and  logistics  re¬ 
quirements  deficiencies; 

(7)  Assess  the  system's  readiness  to  en¬ 
ter  full-rate  production. 

12.5  OT&E  DURING 
THE  PRODUCTION  AND 
DEPLOYMENT  PHASE 
(MS  III  to  MS  IV) 

After  the  MS  III  decision,  the  emphasis 
shifts  towards  procuring  production  quan¬ 
tities,  repairing  hardware  deficiencies, 
managing  changes,  and  phasing  in  full  lo¬ 
gistics  support.  During  initial  deployment 
of  the  system,  the  OT&E  agency  and/or  the 
user  may  perform  follow-on  test  and  evalu¬ 
ation  (FOT&E)  to  refine  the  effectiveness 
and  suitability  estimates  made  during  ear¬ 
lier  OT&E. 

The  FOT&E  is  performed  with  production 
articles  in  operational  organizations.  It  is 
normally  funded  with  operation  and  main¬ 
tenance  (O&M)  funds.  The  first  FOT&E 
conducted  during  this  phase  may  be  used 
to; 

(1)  Ensure  that  the  production  system 
performs  as  well  as  reported  at  the  MS  III 
review; 

(2)  Demonstrate  expected  performance 
and  reliability  improvements; 

(3)  Ensure  that  the  correction  of  deficien¬ 
cies  identified  during  earlier  testing  have 
been  completed; 

(4)  Evaluate  performance  not  tested  dur¬ 
ing  lOT&E. 

Additional  objectives  of  FOT&E  are  to  vali¬ 
date  the  operational  effectiveness  and  suit¬ 


ability  of  a  modified  system  during  an  op¬ 
erational  assessment  of  the  system  in  new 
environments.  The  FOT&E  may  look  at 
different  platform  applications,  new  tacti¬ 
cal  applications  or  the  impact  of  new  threats. 

12.5.1  FOT&E  of  Integrated  Logistic 
Support 

The  testing  objectives  to  evaluate 
postproduction  logistics  readiness  and  sup¬ 
port  are  to: 

(1)  Assess  the  logistics  readiness  and 
sustainability; 

(2)  Evaluate  the  weapon  support  objec¬ 
tives; 

(3)  Assess  the  implementation  of  inte¬ 
grated  logistics  support  plans; 

(4)  Evaluate  the  capability  of  the  logis¬ 
tics  support  activities; 

(5)  Determine  the  disposition  of  dis¬ 
placed  equipment; 

(6)  Evaluate  the  affordability  and  life- 
cycle  cost  of  the  system. 

12.6  SUMMARY 

Operational  test  and  evaluation  is  that  T&E 
(operational  assessments,  lOT&E  or 
FOT&E)  conducted  to  estimate  a  system's 
operational  effectiveness  and  suitability. 
They  will  identify  needed  modifications; 
provide  information  on  tactics,  doctrine, 
organizations  and  personnel  requirements; 
and  evaluate  the  system's  logistic  support- 
ability.  The  acquisition  program  structure 
should  include  operational  assessments  or 
evaluations  beginning  early  in  the  devel¬ 
opment  cycle  and  continuing  throughout 
the  system's  life  cycle. 
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13 

PROGRAM  MANAGEMENT  OFFICE 
OPERATIONAL  TEST  RESPONSIBILITIES 


13.1  INTRODUCTION 

In  the  government  program  management 
office  (PMO),  there  should  be  a  section 
responsible  for  test  and  evaluation  (T&E). 
Besides  being  responsible  for  development 
test  and  evaluation  (DT&E)  support  to  the 
program  manager,  this  section  should  be 
responsible  for  program  coordination  with 
the  operational  test  and  evaluation  (OT &E) 
agency.  The  offices  of  the  systems  engineer 
or  the  Deputy  for  T&E  may  be  designated 
to  provide  this  support  to  the  program 
manager.  In  some  Services,  responsibilities 
of  the  Deputy  for  (T&E)  include  coordina¬ 
tion  of  test  resources  for  all  phases  of  OT &E. 

13.2  CONTRACT  RESPONSIBILITIES 

The  Deputy  for  T&E  or  his  representative 
ensures  that  certain  sections  of  the  Request 
for  Proposal  (RFP)  contain  sufficient  allow¬ 
ance  for  T&E  support  by  contractors.  This 
applies  whether  the  contract  is  for  a  devel¬ 
opment  item,  a  production  it^  (limited 
production,  such  as  low-rate  initial  pro¬ 
duction  (LRIP)  or  full-rate  production)  or 
the  enhancement/upgrade  of  portions  of  a 
weapons  system.  Where  allowed  within 
the  law,  contractor  support  for  OT&E 
should  be  considered  to  help  resolve  basic 
issues  such  as  data  collection  requirements, 
test  resources,  contractor  test  support  and 
funding. 

In  the  overall  portion  of  the  RFP,  govern¬ 
ment  personnel,  especially  those  in  the 


operational  test  agencies,  must  be  guaran¬ 
teed  access  to  the  contractor's  development 
facilities,  particularly  during  the  DT&E 
Phase.  Government  representatives  must 
be  allowed  to  observe  all  contractor  in- 
house  testing  and  have  access  to  test  data 
and  reports. 

13.2.1  Data  Requirements 

The  contract  must  specify  the  data  the  con¬ 
tractor  will  supply  the  operational  test 
agency  (OTA).  Unlike  DT&E,  the  contrac¬ 
tor  will  not  be  making  the  OT&E  plans, 
procedures  or  reports.  These  documents 
are  the  responsibility  of  the  OTA.  The  PMO 
Deputy  for  T&E  should  include  the  OTA 
on  the  distribution  list  for  all  test  docu¬ 
ments  that  are  of  concern  during  the  DT&E 
phase  of  testing  so  they  will  be  informed  of 
test  item  progress  and  previous  testing.  In 
this  way,  the  OTA  will  be  informed  when 
developing  their  own  test  plans  and  proce¬ 
dures  for  OT&E.  In  fact,  OTA  representa¬ 
tives  should  attend  the  Contract  Data  Re¬ 
quirements  List  (CDRL)  Review  Board  and 
provide  the  PMO  with  a  list  of  the  types  of 
documents  the  OTA  will  need.  The  Deputy 
for  T&E  should  coordinate  the  test  sections 
of  this  data  list  with  the  OTA  and  indicate 
concerns  at  that  meeting.  All  contractor  test 
reports  should  be  made  available  to  the 
OTA.  In  return,  the  Deputy  for  T&E  must 
ensure  that  he  is  informed  of  all  OTA  activi- 
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ties,  understands  their  test  procedures  and 
plans  and  receives  their  test  reports.  Unlike 
DT&E,  the  PMO  Deputy  for  T&E  will  not 
have  report  or  document  approval  author¬ 
ity  as  he  does  over  contractor  documenta¬ 
tion.  The  Deputy  for  T&E  is  always  respon¬ 
sible  for  keeping  the  program  manager 
informed  of  OT&E  results. 

13.2.2  Test  Schedule 

Another  important  early  activity  the  Deputy 
for  T&E  must  accomplish  is  to  coordinate 
the  OT&E  test  schedule.  Since  the  contrac¬ 
tor  may  be  required  to  provide  support,  the 
OT&E  test  support  may  need  to  be  contrac¬ 
tually  agreed  upon  before  contract  award. 
Sometimes,  the  Deputy  for  T&E  can  formu¬ 
late  a  strawman  schedule  (based  on  previ¬ 
ous  experience)  and  present  this  schedule 
to  the  operational  test  representative  at  the 
initial  test  planning  working  group  for  re¬ 
view;  or  he  can  contact  the  OTA  and  ar¬ 
range  a  meeting  to  discuss  the  new  pro¬ 
gram.  In  the  meeting,  time  requirements 
envisioned  by  OTA  can  be  discussed.  Input 
from  that  meeting  then  goes  into  the  RFP 
and  to  the  program  manager.  The  test  sched¬ 
ule  must  allow  time  for  DT&E  testing  and 
OT&E  testing  if  testing  is  not  combined  or 
test  assets  are  limited.  Before  set-up  of  ini¬ 
tial  operational  test  and  evaluation  (lOT&E), 
certification  of  readiness  for  lOT&E  may 
require  a  time  gap  for  review  of  DT&E  test 
results  and  refurbishment  or  corrections  of 
deficiencies  discovered  during  DT&E,  etc. 
The  test  schedule  for  DT&E  should  not  be 
so  "success-oriented"  that  the  lOT&E  test 
schedule  is  adversely  impacted,  not  allow¬ 
ing  enough  time  for  adequate  operational 
testing  or  the  reporting  of  lOT&E  results. 
For  example,  if  the  DT&E  schedule  slips  six 
months,  the  OT&E  schedule  and  milestone 
decision  should  slip  also.  The  lOT&E  should 
not  be  shortened  just  to  make  a  milestone 
decision  date. 


13.2.3  Contractor  Support 

The  Deputy  for  T&E  provides  all  T&E  in¬ 
put  to  the  RFP /SOW;  he  must  determine, 
before  the  beginning  of  the  program  acqui¬ 
sition  phase,  whether  the  contractor  will  be 
involved  in  supporting  OT&E  and,  if  so,  to 
what  extent.  According  to  Title  1 0,  USC,  the 
system  contractor  can  only  be  involved  in 
the  conduct  of  lOT&E  if,  once  the  item  is 
fielded,  tactics  and  doctrine  say  the  con¬ 
tractor  will  be  providing  support  or  operat¬ 
ing  that  item  during  combat.  If  not,  no 
system  contractor  support  is  allowed  dur¬ 
ing  OT&E.  Before  lOT&E,  however,  the 
contractor  may  be  tasked  with  providing 
training,  training  aids  and  handbooks  to 
Service  training  cadre  so  they  can  train  the 
lOT&E  users  and  maintenance  personnel. 
In  addition,  the  contractor  must  be  required 
to  provide  sufficient  spare  parts  for  the 
operational  maintenance  personnel  to 
maintain  the  test  item  while  undergoing 
operational  testing.  These  support  items 
must  be  agreed  upon  by  the  PMO  and  OTA 
and  must  contractually  bind  the  contrac¬ 
tor.  If,  however,  the  contractor  will  be  re¬ 
quired  to  provide  higher-level  maintenance 
of  the  item  for  the  duration  of  the  lOT&E, 
data  collection  on  those  functions  will  be 
delayed  until  a  subsequent  follow-on  op¬ 
erational  test  and  evaluation  (FOT&E). 

13.2.4  OT&E  Funding 

The  Deputy  for  T&E  provides  the  program 
manager  estimates  of  PMO  test  program 
costs  to  conduct  lOT&E.  This  funding  in¬ 
cludes  contractor  and  government  test  sup¬ 
port  for  which  the  program  office  directly 
or  indirectly  will  be  responsible.  Since  Ser¬ 
vice  OTAs  fund  differently,  program  office 
funding  for  conducting  OT&E  varies.  The 
Deputy  for  T&E  must  determine  these 
costs  and  inform  the  program  manager. 
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13.2.5  Statement  of  Work 

One  of  the  most  important  documents  re¬ 
ceiving  input  from  the  Deputy  for  T&E  is 
the  Statement  of  Work  (SOW).  He  must 
outline  all  required  or  anticipated  contrac¬ 
tor  support  for  DT&E  and  OT&E.  This  docu¬ 
ment  outlines  data  requirements,  contrac¬ 
tor-conducted  or  supported  testing,  gov- 
e’^.ment  involvement  (access  to  contractor 
data,  tests  and  results),  operational  test  sup¬ 
port,  and  any  other  specific  test  require¬ 
ments  the  contractor  will  be  tasked  to  per¬ 
form  during  the  duration  of  the  contract. 

13.3  TEST  AND  EVALUATION 
MASTER  PLAN 

The  Test  and  Evaluation  Master  Plan 
(TEMP)  should  be  updated  regularly  by 
the  OTA.  The  Deputy  for  T&E  is  respon¬ 
sible  for  managing  the  TEMP  throughout 
the  test  program.  The  operational  test 
agency  usually  is  tasked  to  complete  the 
operational  test  section  of  the  TEMP  and 
outline  their  proposed  test  program  through 
all  phases  of  OT&E.  It  is  important  to  keep 
the  TEMP  updated  regularly  so  that  test 
organizations  involved  in  OT&E  under¬ 
stand  the  scope  of  their  test  support.  Fur¬ 
ther,  if  any  upgrades,  improvements  or 
enhancements  to  the  fielded  weapon  sys¬ 
tem  occur,  the  TEMP  must  be  updated  or  a 
new  one  created  to  outline  new  develop¬ 
ment  test  (DT)  and  operational  test  (OT) 
requirements. 

13.4  PHASES  OF  OPERATIONAL  TEST 

For  lOT&E,  the  Deputy  for  T&E  must  en¬ 
sure  the  contract  portions  adequately  cover 
the  scope  of  testing  as  outlined  by  the  op¬ 
erational  test  agency.  The  program  office 
may  want  to  provide  an  observer  to  repre¬ 
sent  the  Deputy  for  T&E  during  the  actual 
testing.  The  Deputy  for  T&E  involvement 


in  lOT&E  will  be  to  monitor  and  coordi¬ 
nate;  he  will  keep  the  program  manager 
informed  of  progress  and  problems  that 
arise  during  testing  and  will  monitor  re¬ 
quired  PMO  support  to  the  test  organiza¬ 
tion.  Also,  enough  LRIP  items  must  be 
manufactured  to  run  a  complete  and  ad¬ 
equate  OT&E  program.  For  problems  re¬ 
quiring  program  office  action,  the  Deputy 
for  T&E  will  be  the  point  of  contact. 

The  Deputy  for  T &E  will  be  concerned  with 
lOT&E  of  the  LRIP  units  after  a  limited 
number  are  produced.  The  lOT&E  must  be 
closely  monitored  so  that  a  full-rate  pro¬ 
duction  decision  can  be  made.  As  in  the 
operational  assessments,  the  Deputy  for 
T&E  will  be  monitoring  test  procedures 
and  results  and  keeping  the  program  man¬ 
ager  informed.  If  the  item  does  not  succeed 
during  lOT&E,  a  new  process  of  DT&E  or  a 
modification  may  result;  and  the  Deputy 
for  T&E  will  be  involved  (as  in  any  new 
programs  inception).  If  the  item  passes 
lOT&E  testing  and  is  produced  at  full  rate, 
the  Deputy  for  T&E  will  be  responsible  for 
ensuring  that  testing  of  those  production 
items  is  adequate  to  ensure  that  the  end- 
items  physically  and  functionally  resemble 
the  development  items. 

During  FOT&E,  the  Deputy  for  T&E  moni¬ 
tors  the  testing;  the  contractor  is  usually 
not  involved.  The  Deputy  for  T&E  should 
receive  any  reports  generated  by  the  opera¬ 
tional  testers  during  this  time.  Any  defi¬ 
ciencies  noted  during  FOT&E  should  be 
evaluated  by  the  PMO,  which  may  decide 
to  incorporate  upgrades,  enhancements  or 
additions  to  the  current  system.  If  the  pro¬ 
gram  manager  and  the  engineering  section 
of  the  program  office  design  or  develop 
modifications  that  are  incorporated  into 
the  weapon  system  design,  additional 
FOT&E  may  be  required. 
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13.5  FOT&E  FOR  UPGRADES, 
ENHANCEMENTS,  ADDITIONS 

Once  a  weapon  system  is  fielded,  portions 
of  that  system  may  become  obsolete,  inef¬ 
fective  or  deficient  and  may  need  replac¬ 
ing,  upgrading  or  enhancing  to  ensure  the 
weapon  system  meets  current  and  future 
requirements.  The  Deputy  lor  T&E  plays  a 
vital  role  in  this  process.  Modifications  to 
existing  weapon  systems  may  be  managed 
as  an  entire  newly  acquired  weapon  sys¬ 
tem.  However,  since  these  are  changes  to 
existing  systems,  the  Deputy  for  T&E  is 
responsible  for  determining  if  these  en¬ 
hancements  degrade  the  existing  system, 
are  compatible  with  its  interfaces  and  func¬ 
tions  and  whether  nondevelopment  items 
(NDIs)  require  retest  or  the  entire  weapon 
system  needs  reverification.  The  Deputy 
for  T&E  must  plan  the  test  program's  fund¬ 
ing,  schedule,  test  program  and  contract 
provisions  with  these  items  in  mind.  A  new 
TEMP  may  have  to  be  generated  or  the 
original  weapon  system  TEMP  modified 
and  recoordinated  with  the  test  organiza¬ 
tions.  The  design  of  the  DT&E  and  FOT&E 
program  usually  requires  coordination  with 
the  engineering,  contracting  and  program 
management  sections  of  the  program  of¬ 
fice. 

13.6  TEST  RESOURCES 

During  all  phases  of  OT,  the  Deputy  for 
T&E  must  coordinate  with  the  operational 
testers  to  ensure  they  have  the  test  articles 


needed  to  accomplish  their  mission.  Test 
resources  will  be  either  contractor  provided 
or  government  provided.  The  contractor 
resources  must  be  covered  in  the  contract, 
whether  in  the  development  contract  or  the 
production  contract.  Government  test  re¬ 
sources  needed  are  determined  by  the  op¬ 
erational  testers.  They  usually  coordinate 
the  test  ranges,  test  support  and  the  user 
personnel  for  testing.  The  program  man¬ 
ager  programs  funding  for  his  support  of 
OT.  Funding  for  Navy  operational  evalua¬ 
tion  (OPEVAL)  is  identified  in  the  TEMP 
and  funded  in  the  PMO's  budget.  Other 
Services  allow  the  OTAs  to  develop  and 
manage  their  own  budget  for  operational 
testing.  The  OTAs  then  obligate  funds  for 
test  ranges,  instrumentation,  etc.,  accord¬ 
ing  to  theii  operational  test  plans. 

13.7  SUMMARY 

The  PMO  should  be  proactive  in  its  rela¬ 
tions  with  the  Service  operational  testing 
agency.  There  are  many  opportunities  to 
educate  the  OTA  on  system  characteristics 
and  expected  performance.  Early  OTA  in¬ 
put  to  design  considerations  and  require¬ 
ments  clarification  can  reduce  downstream 
surprises.  Operational  testing  is  an  essen¬ 
tial  component  of  the  development  and 
decision-making  process.  It  can  be  used  to 
facilitate  system  development  or  may  be¬ 
come  an  impediment.  In  many  cases,  the 
PMO  attitude  toward  operational  testing 
and  the  OTA  will  influence  which  role  the 
OTA  assumes. 
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•  Understand  the  policies 

•  Organize  for  T&E 

•  Keep  system  requirements  documents  current 

•  Agonize  over  system  thresholds 

•  Work  closely  with  the  operational  test  director 

•  Don’t  forget  about  operational  suitability 

•  Make  final  DT&E  a  rehearsal  for  lOT&E 

•  Prepare  interfacing  systems  for  your  lOT&E 

•  Manage  software  testing  closely 

•  Track  availability  of  test  resources  and  test 
support  personnel/facilities 

Source:  NAVSEA  T&E  Office 


Figure  13-1.  Lessons  Learned  From  OT&E 
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IV 

MODULE 

Test  and  Evaluation 
Planning 


Many  program  managers  face  several  test 
and  evaluation  issues  that  must  be  resolved 
to  get  their  particular  weapon  system  tested 
and  ultimately  fielded.  These  issues  may 
include  modeling  and  simulation  support, 
combined  and  concurrent  testing,  test  re¬ 
sources,  survivability  and  lethality  testing, 
multi-Service  testing,  or  international  T &E. 
Each  issue  presents  a  imique  set  of  chal¬ 
lenges  for  the  program  manager  when  he 
develops  the  integrated  strategy  for  the  test 
and  evaluation  program. 


14 

EVALUATION 


14.1  INTRODUCTION 

This  chapter  describes  the  evaluation  por¬ 
tion  of  the  test  and  evaluation  process.  It 
stresses  the  importance  of  establishing  and 
maintaining  a  clear  audit  trail  from  system 
requirements  through  critical  issues,  evalu¬ 
ation  criteria,  test  objectives  and  measures 
of  effectiveness  to  the  evaluation.  The  im¬ 
portance  of  the  use  of  data  from  all  sources 
is  discussed  as  are  the  differences  in  ap¬ 
proaches  to  evaluating  technical  and  op¬ 
erational  data. 

14.2  DIFFERENCE  BETWEEN 
"TEST"  AND  "EVALUATION" 

The  following  distinction  has  been  made 
between  the  functions  of  "test"  and  "evalu¬ 
ation": 

While  the  terms  "test"  and  "evalua¬ 
tion"  are  most  often  found  together, 
they  actually  denote  clearly  distin¬ 
guishable  functions  in  the  RDT&E  [re¬ 
search,  development,  test  and  evalua¬ 
tion]  process. 

"Test"  denotes  the  actual  testing  of  hard¬ 
ware/software  -models,  protot)qjes,  pro¬ 
duction  equipment,  computer  programs 
— to  obtain  data,  including  software,  valu¬ 
able  in  developing  new  capabilities,  manag¬ 
ing  the  process,  or  making  decisions  on  the 
allocation  of  resources. 

"Evaluation"  denotes  the  process  whereby 
data  are  logically  assembled  and  analyzed 


to  aid  in  making  systematic  decisions.  (Ref¬ 
erence  10) 

To  summarize,  evaluation  is  "the  review 
and  analysis  of  qualitative  or  quantitative 
data  obtained  from  design  review,  hard¬ 
ware  inspection,  testing  or  operational  us¬ 
age  of  equipment,"  (Reference  2). 

14.3  THE  EVALUATION 
PROCESS 

The  evaluation  process  requires  a  broad 
analytical  approach  with  careful  focus  on 
the  development  of  an  overall  test  and 
evaluation  (T&E  plan  that  will  provide 
timely  answers  to  critical  issues  and  ques¬ 
tions  required  by  decision  authorities 
throughout  all  the  acquisition  phases. 
Evaluations  should  focus  on  critical  sys¬ 
tem  characteristics;  i.e.,  "those  design  fea¬ 
tures  that  determine  how  well  the  pro¬ 
posed  concept  or  system  will  function  in  its 
intended  operational  environment,"  (4-C, 
DODI  5000.2). 

A  fvmctional  block  diagram  of  a  generic 
(i.e.,  not  Service-specific)  evaluation  pro¬ 
cess  is  shown  in  Figure  14-1.  The  process 
begins  with  the  identification  of  a  defi¬ 
ciency  or  need  and  the  documentation  of 
an  operational  requirement.  It  continues 
with  the  identification  of  critical  issues  that 
must  be  addressed  to  determine  the  degree 
to  which  the  system  meets  user  require¬ 
ments.  Objectives  and  thresholds  must  then 
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Figure  14-1.  Functional  Block  Diagram  of  the  Evaluation  Process 


be  established  to  define  required  perfor¬ 
mance  or  supportability  parameters  and  to 
evaluate  progress  in  reaching  them.  Test 
and  evaluation  analysts  then  decompose 
the  issues  into  measurable  test  elements, 
conduct  the  necessary  testing,  review  and 
analyze  the  test  data,  weigh  the  test  results 
against  the  evaluation  criteria,  and  prepare 
an  evaluation  report  for  the  decision  au¬ 
thorities. 

14.4  ISSUES  AND  CRITERIA 

Issues  are  questions  regarding  a  system 
that  require  answers  during  the  acquisi¬ 
tion  process.  Those  answers  may  be  needed 
to  aid  in  the  development  of  an  acquisition 
strategy,  to  refine  performance  »'equire- 
ments  and  designs  or  to  support  milestone 
decision  reviews.  Evaluation  criteria  are 
the  standards  by  which  accomplishments 
of  required  technical  and  operational  effec¬ 
tiveness  and/or  suitability  characteristics 
or  resolution  of  operational  issues  may  be 
assessed.  The  evaluation  program  may  be 
constructed  using  a  structured  approach 
identifying  each  issue. 

(1)  Issue  —  a  statement  of  the  question 
to  be  answered; 

(2)  Scope  —  detailed  conditions  and 
range  of  conditions  that  will  guide  the  T &E 
process  for  this  issue; 

(3)  Criteria  —  quantitative  or  qualita¬ 
tive  standards  that  will  answer  the  issue; 

(4)  Rationale  —  full  justification  to  sup¬ 
port  the  selected  criteria. 

(See  Appendices  for  example.) 

14.4.1  System  Program  Issues/ 

Critical  Issues 

System  program  issues  often  consist  of  a 
hierarchy  of  critical  issues  and  less  signifi¬ 


cant  issues.  Critical  issues  are  those  ques¬ 
tions  relating  to  a  system's  operational, 
technical,  support  or  other  capability.  These 
issues  must  be  answered  before  the 
system's  overall  worth  can  be  estimated/ 
evaluated,  and  they  are  of  primary  impor¬ 
tance  to  the  decision  authority  in  allowing 
the  system  to  advance  to  the  next  acquisi¬ 
tion  phase.  Critical  issues  in  the  TEMP  may 
be  derived  from  the  critical  system  charac¬ 
teristics  found  in  the  operational  require¬ 
ment  document.  The  system  requirements 
and  baseline  documentation  will  provide 
many  of  the  performance  parameters  re¬ 
quired  to  de\  elop  the  hierarchy  of  issues. 

14.4.2  Evaluation  Issues 

Evaluation  issues  are  those  addressed  in 
technical  or  operational  evaluations  dur¬ 
ing  the  acquisition  process.  Evaluation  is¬ 
sues  can  be  separated  into  technical  or 
operational  issues  and  addressed  in  the 
Test  and  Evaluation  Master  Plan  (TEMP). 

Technical  issues  primarily  concern  techni¬ 
cal  parameters  or  characteristics  and  engi¬ 
neering  specifications  normally  assessed 
in  development  testing.  Operational  issues 
concern  effectiveness  and  suitability  char¬ 
acteristics  for  functions  to  be  performed  by 
equipment/personnel.  They  address  the 
system's  operational  performance  when 
examined  in  a  realistic  operational  mission 
environment.  Evaluation  issues  are  an¬ 
swered  by  whatever  means  necessary 
(analysis /survey,  modeling,  simulation, 
demonstration  or  testing)  to  resolve  the 
issue.  Issues  requiring  test  data  are  further 
referred  to  as  test  issues. 

14.4.3  Test  Issues 

Test  issues  are  a  subset  of  evaluation  issues 
and  address  areas  of  uncertainty  that  re¬ 
quire  test  data  to  resolve  the  issue  ad¬ 
equately.  Test  issues  can  be  separated  into 
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technical  issues  that  are  addressed  by  the 
development  test  and  evaluation  (DT&E) 
community  and  operational  issues  that  are 
addressed  by  the  operational  test  and  evalu¬ 
ation  (OT&E)  community.  Test  issues  may 
be  divided  into  critical  and  noncritical  cat¬ 
egories.  All  critical  T&E  issues,  objectives, 
methodologies  and  evaluation  criteria 
should  be  defined  during  the  initial  estab¬ 
lishment  of  an  acquisition  program.  Criti¬ 
cal  issues  are  documented  in  the  TEMP. 
1  hese  evaluation  issues  serve  to  define  the 
testing  required  for  each  phase  of  the  ac¬ 
quisition  process  and  serve  as  the  structure 
to  guide  the  testing  program  so  these  data 
may  be  compared  against  performance  cri¬ 
teria. 

14.4.4  Criteria 

Criteria  are  statements  of  a  system's  re¬ 
quired  technical  performance  and  opera¬ 
tional  effectiveness,  suitability  and  sup- 
portability.  Criteria  are  often  expressed  as 
"objectives  and  thresholds."  (Some  Ser¬ 
vices,  however,  specify  performance  and 
supportability  requirements  exclusively  in 
terms  of  thresholds  and  avoid  the  use  of  the 
concept  of  objectives.)  These  performance 
measurements  provide  the  basis  for  col¬ 
lecting  data  used  to  evaluate/answer  test 
issues. 

Criteria  must  be  ;  ';rian  ibiguous  and  assess¬ 
able  whether  stated  qualitatively  or  quan¬ 
titatively.  They  may  compare  the  mission 
performance  of  the  new  system  to  the  one 
being  replaced,  compare  the  new  system  to 
a  predetermined  standard,  or  compare 
mission  performance  results  using  the  new 
system  to  not  having  the  system.  Criteria 
are  the  final  values  deemed  necessary  by 
the  user.  As  such,  they  should  be  devel¬ 
oped  in  close  coordination  with  the  system 
user,  other  testers  and  specialists  in  all 
other  areas  of  operational  effectiveness  and 
suitability.  These  values  may  be  changed 


as  systems  develop  and  associated  testing 
and  evaluation  proceed.  Every  issue  should 
have  at  least  one  criteria  that  is  a  concise 
measure  of  the  function.  Values  must  be 
realistic  and  achievable  within  the  state  of 
the  art  of  engineering  technology.  A  quan¬ 
titative  or  qualitative  criterion  should  have 
a  clear  definition,  free  of  ambiguous  or 
imprecise  terminology,  such  as  "adequate," 
"sufficient"  or  "acceptable." 

14.4.4.1  Test  and  Thresholds 
and  Objectives 

An  operational  requirement  document 
(ORD)  threshold  performance  parameter 
lists  a  minimally  acceptable  requirement  or 
a  minimally  acceptable  level  of  performance 
required  by  a  test  article  or  system  to  pro¬ 
vide  a  system  capability  that  will  satisfy  the 
validated  mission  need.  Thresholds  are 
stated  quantitatively  whenever  possible. 
Specification  of  minimally  acceptable  per¬ 
formance  in  measurable  parameters  is  es¬ 
sential  to  selecting  appropriate  measures 
of  effectiveness,  which,  in  turn,  heavily 
influence  test  design.  Thresholds  are  of 
value  only  when  they  are  testable;  i.e.,  ac¬ 
tual  performance  can  be  measured  against 
them.  The  function  of  T&E  is  to  verify  the 
attainment  of  required  thresholds.  As  stated 
in  OPNAVINST  5000.42C,  "T&E  is  the 
major  control  mechanism  of  the  acquisition 
process.  Programs  advance  from  one  phase 
to  the  next,  not  by  the  calendar  or  planned 
schedule,  but  by  actual  achievement  of 
present  thresholds,  verified  by  T&E."  (Ref¬ 
erence  69) 

Objectives  are  levels  of  performance  (estab¬ 
lished  by  the  user)  above  the  threshold 
that,  if  achieved,  will  provide  measurable 
benefits  of  additional  operational  capabil¬ 
ity,  operations  and  support.  Objectives  are 
not  normally  addressed  by  the  operational 
tester,  whose  primary  concern  is  the  re¬ 
quirement. 
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Going  into  Milestone  II,  thresholds  and 
objectives  are  expanded  along  with  the 
identification  of  more-detailed  and  refined 
performance  capabilities  and  characteris¬ 
tics  resulting  from  trade-off  studies  and 
testing  conducted  during  the  Demonstra¬ 
tion  and  Validation  Phase.  Along  with  the 
ORD,  they  should  remain  relatively  stable 
through  production. 

14.5  MEASURES  OF  EFFECTIVENESS 

Requirements,  thresholds  and  objectives 
established  in  early  program  documenta¬ 
tion  form  the  basis  forevaluation  criteria.  If 
program  documentation  is  incomplete,  the 
tester  may  have  to  develop  evaluation  cri¬ 
teria  in  the  absence  of  specific  require¬ 
ments.  Evaluation  criteria  are  associated 
with  objectives,  subobjectives  and  mea¬ 
sures  of  effectiveness  (MOEs)  .For  example, 
an  MOE  (e.g.,  airspeed)  may  have  an  asso¬ 
ciated  evaluation  criterion  (e.g.,  450  knots) 
against  which  the  actual  performance  (e.g., 
425  knots)  is  compared  to  arrive  at  a  rating. 
An  MOE  of  a  system  is  a  parameter  that 
evaluates  the  capacity  of  the  system  to 
accomplish  its  assigned  missions  under  a 
given  set  of  conditions.  They  are  important 
because  they  determine  how  test  results 
will  be  judged;  and,  since  test  planning  is 
directed  toward  obtaining  these  measures, 
it  is  important  that  they  be  defined  early. 
Generally,  the  resolution  of  each  critical 
issue  is  in  terms  of  the  evaluation  of  some 
MOE  (Reference  1 16).  In  this  case,  the  oper¬ 
ating,  implementing,  and  supporting  com¬ 
mands  must  agree  with  the  criteria  before 
the  test  organization  makes  use  of  them  in 
assessing  test  results.  Ensuring  that  MOEs 
can  be  related  to  the  user's  operational 
requirements  is  an  important  consideration 
when  identifying  and  establishing  evalua¬ 
tion  criteria.  Testers  must  ensure  that  evalu¬ 
ation  criteria  and  MOEs  are  updated  if 
requirements  change.  Measures  of  effec¬ 
tiveness  should  be  so  specific  that  the 


system's  effectiveness  during  developmen¬ 
tal  and  operational  testing  can  be  assessed 
using  the  same  effectiveness  criteria  as  the 
Cost  and  Operational  Effectiveness  Analy¬ 
sis  (4-E,  DODI  5000.2). 

14.6  EVALUATION  PLANNING 

14.6.1  Evaluation  Planning 
Techniques 

Evaluation  planning  is  an  iterative  process 
that  requires  formal  and  informal  analyses 
of  system  operation  (e.g.,  threat  environ¬ 
ment,  system  design,  tactics  and 
interoperability).  Techniques  that  have 
been  proven  effective  in  evaluation  plan¬ 
ning  include:  process  analysis,  design  or 
engineering  analysis,  matrix  analysis  and 
dendritic  analysis  (Reference  61). 

14.6.1.1  Process  Analysis 
Techniques 

Process  analysis  techniques  consist  of  think¬ 
ing  through  how  the  system  will  be  used  in 
a  variety  of  environments,  threats,  mis¬ 
sions  and  scenarios  in  order  to  understand 
the  events,  actions,  situations  and  results 
that  are  expected  to  occur.  This  technique 
aids  in  the  identification  and  clarification 
of  appropriate  MOEs,  test  conditions  and 
data  requirements. 

14.6.1.2  Design/Engineering 
Analysis  Techniques 

Design  or  engineering  analysis  techniques 
are  used  to  examine  all  mechanical  or  func¬ 
tional  operations  that  the  system  has  been 
designed  to  perform.  These  techniques  in¬ 
volve  a  systematic  exploration  of  the 
system's  hardware  and  software  compo¬ 
nents,  purpose,  performance  bounds,  man¬ 
power  and  personnel  considerations, 
known  problem  areas  and  impact  on  other 
components.  Exploration  of  the  way  a  sys- 
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tern  operates  compared  to  intended  perfor¬ 
mance  functions  often  identifies  issues, 
MOEs,  specific  data,  test  events  and  re¬ 
quired  instrumentation. 

14.6.1.3  Matrix  Analysis 
Techniques 

Matrix  analysis  techniques  are  useful  for 
analyzing  any  situation  where  two  classifi¬ 
cations  must  be  cross-referenced.  For  ex¬ 
ample,  a  matrix  of  "types  of  data"  vs.  "means 
of  data  collection"  can  reveal  not  only  types 
of  data  having  no  planned  means  of  collec¬ 
tion  but  also  redundant  or  backup  collec¬ 
tion  systems.  Matrix  techniques  are  useful 
as  checklists,  as  organizational  tools  or  as  a 
way  of  identifying  and  characterizing  prob¬ 
lem  areas.  Matrix  techniques  are  effective 
for  tracing  a  system's  operational  require¬ 
ments  through  contractual  specification 
documents,  issues  and  criteria  to  sources  of 
individual  data  or  specific  test  events. 

14.6.1.4  Dendritic  Analysis 
Techniques 

Dendritic  analysis  techniques  are  an  effec¬ 
tive  way  of  decomposing  critical  issues  to 
the  point  where  actual  data  requirements 
and  test  measurements  can  be  identified.  In 
these  techniques,  issues  are  successively 
broken  down  into  objectives,  measures  of 
effectiveness,  measures  of  performance  and 
data  requirements  in  a  root-like  structure 
as  depicted  in  Figure  14-2.  In  this  approach, 
objectives  are  used  to  clearly  express  the 
broad  aspects  of  T&E  related  to  the  critical 
issues  and  the  overall  purpose  of  the  test. 
Measures  of  effectiveness  are  developed  as 
subsets  of  the  objectives  and  are  designed 
to  treat  specific  and  addressable  parts  of  the 
objectives.  Each  MOE  is  traceable  as  a  di¬ 
rect  contributor  to  one  objective  and, 
through  it,  is  identifiable  as  a  direct  con¬ 
tributor  to  addressing  one  or  more  critical 
issues  (Reference  83).  Each  test  objective 


and  MOE  is  also  linked  to  one  or  more 
measures  of  performance  (quantitative  or 
qualitative  measures  of  system  perfor¬ 
mance  under  specified  conditions)  that,  in 
turn,  are  tied  to  specific  data  elements.  The 
dendritic  approach  has  become  a  standard 
military  planning  technique. 

14.6.2  Sources  of  Data 

As  evaluation  and  analysis  planning  ma¬ 
tures,  focus  turns  toward  identifying  data 
sources  as  a  means  for  obtaining  each  data 
element.  Initial  identification  tends  to  be 
generic  such  as:  engineering  study,  simu¬ 
lation,  modeling  or  contractor  test.  Later 
identification  reflects  specific  studies,  mod¬ 
els  and/or  tests.  A  data  source  matrix  is  a 
useful  planning  tool  to  show  where  data 
are  expected  to  be  obtained  during  the  T&E 
of  the  system. 

There  are  many  sources  of  data  that  can 
contribute  to  the  evaluation.  Principal 
sources  include:  studies  and  analyses,  mod¬ 
els,  simulations,  war  games,  contractor  test¬ 
ing,  development  testing,  operational  test¬ 
ing  and  comparable  systems. 

14.7  EVALUATING 

DEVELOPMENT 

AND  OPERATIONAL  TESTS 

Technical  and  operational  evaluations 
employ  different  techniques  and  have  dif¬ 
ferent  evaluation  criteria.  Development  test 
and  evaluation  is  often  considered  techni¬ 
cal  evaluation  while  OT&E  addresses  the 
operational  aspects  of  a  system.  Technical 
evaluation  deals  primarily  with  instru¬ 
mented  tests  and  statistically  valid  data. 
An  operational  evaluation  deals  with  op¬ 
erational  realism  and  the  combat  uncer¬ 
tainties  (Reference  76).  Development  test 
and  evaluation  uses  technical  criteria  for 
evaluating  system  performance.  These  cri¬ 
teria  are  usually  parameters  that  can  be 
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DATA  ELEMENT  1 
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Rgure  14-2.  Dendritic  Approach  to  Test  and  Evaluation 


measured  during  controlled  DT&E  tests. 
They  are  particularly  important  to  the  de¬ 
veloping  organization  and  the  contractor 
but  are  of  less  interest  to  the  independent 
operational  tester.  The  operational  tester 
focuses  on  issues  such  as  demonstrating 
target  acquisition  at  useful  ranges,  air  su¬ 
periority  in  combat,  or  the  probability  of 
accomplishing  a  given  mission.  For  ex- 
axnple,  during  DT&E,  firing  may  be  con¬ 
ducted  on  a  round-by-round  basis  with 
each  shot  designed  to  test  an  individual 
specification  or  parameter  with  other  pa¬ 
rameters  held  constant.  Such  testing  is  de¬ 
signed  to  measure  the  technical  perfor¬ 
mance  of  the  system.  In  contrast,  in  OT&E 
proper  technical  performance  regarding 
individual  specifications/parameters  isde- 
emphasized  and  the  environment  is  less 
controlled.  The  OT&E  authority  must  as¬ 
sess  whether,  given  this  technical  perfor¬ 
mance,  the  weapon  system  is  operationally 
effective  and  operationally  suitable  when 
employed  under  realistic  combat  (with  op¬ 
posing  force)  and  environmental  condi¬ 
tions  by  typical  personnel. 

The  emphasis  in  development  test  (DT)  is 
strictly  on  the  use  of  quantitative  data  to 
verify  attainment  of  technical  specifications. 
Quantitative  data  are  usually  analyzed  us¬ 
ing  some  form  of  statistics.  Qualitative  data 
takes  on  increasing  importance  in  OT&E 
when  effectiveness  and  suitability  issues 
are  being  explored.  Many  techniques  are 
used  to  analyze  qualitative  data.  They  range 
from  converting  expressions  of  preference 
or  opinion  into  numerical  values  to  estab¬ 
lishing  a  consensus  by  committee.  For  ex¬ 
ample,  a  committee  may  assign  values  to 
parameters  such  as  “feel,"  "ease  of  use," 
"friendliness  to  the  user,"  and  "will  the 
user  want  to  use  it,"  on  a  scale  of  1-to-lO. 
Care  should  be  exercised  in  the  interpreta¬ 
tion  of  the  results  of  qualitative  evalua¬ 
tions.  For  instance,  when  numbers  are 


assigned  to  average  evaluations  and  their 
standard  deviations,  meanings  will  differ 
from  quantitative  data  averages  and  stan¬ 
dard  deviations. 

14.7.1  Technical  Evaluation 

The  Services'  materiel  development  orga¬ 
nizations  are  usually  responsible  for  over¬ 
sight  of  all  aspects  of  DT&E  including  the 
technical  evaluation.  The  objectives  of  a 
technical  evaluation  are: 

•  To  assist  the  developers  by  providing 
information  relative  to  technical  perfor¬ 
mance;  qualification  of  components;  com¬ 
patibility,  inter-operability,  vulnerability, 
lethality,  transportability,  reliability,  avail¬ 
ability  and  maintainability  (RAM);  man¬ 
power  and  personnel;  system  safety;  inte¬ 
grated  logistics  support;  correction  of  defi¬ 
ciencies;  accuracy  of  environmental 
documentation;  and  refinement  of  require¬ 
ments; 

•  To  ensure  the  effectiveness  of  the  manu¬ 
facturing  process  of  equipment  and  proce¬ 
dures  through  production  qualification 
T&E; 

•  To  confirm  readiness  for  operational 
test  (OT)  by  ensuring  that  the  system  is 
stressed  beyond  the  levels  expected  in  the 
OT  environment; 

•  To  provide  information  to  the  decision 
authority  at  each  decision  point  regarding 
a  system's  technical  performance  and  readi¬ 
ness  to  proceed  to  the  next  phase  of  acquisi¬ 
tion; 

•  To  determine  the  system's  operability 
in  the  required  climatic  and  realistic  battle¬ 
field  environments  to  include  natural,  in¬ 
duced,  and  countermeasure  environments 
(Reference  54). 
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14.7.2  Operational  Evaluation 

The  independent  OT&E  authority  is  re¬ 
sponsible  for  the  operational  evaluation. 
The  objectives  of  an  operational  evaluation 
are: 

•  To  assist  the  developers  by  providing 
information  relative  to  operational  perfor¬ 
mance;  doctrine,  tactics,  training,  logistics; 
safety;  survivability;  manpower,  technical 
publications;  RAM;  correction  of  deficien¬ 
cies;  accuracy  of  environmental  docu¬ 
mentation;  and  refinement  of  requirements; 

•  To  assist  decision-makers  ensure  that 
only  systems  that  are  operationally  effec¬ 
tive  and  suitable  are  delivered  to  the  oper¬ 
ating  forces; 

•  To  provide  information  to  the  decision 
authority  at  each  decision  point  as  to  a 
system's  operational  effectiveness,  suitabil¬ 
ity  and  readiness  to  proceed  to  the  next 
phase  of  acquisition; 

•  To  assess,  from  the  user's  viewpoint,  a 
system's  desirability,  considering  systems 


already  fielded,  and  the  benefits  or  bur¬ 
dens  associated  with  the  system  (Refer¬ 
ence  84). 

14.8  SUMMARY 

A  primary  consideration  in  identifying  in¬ 
formation  to  be  generated  by  an  evalua¬ 
tion  program  is  having  a  clear  understand¬ 
ing  of  the  decisions  the  information  will 
support.  The  importance  of  structuring 
the  T&E  program  to  support  the  resolution 
of  critical  issues  cannot  be  overempha¬ 
sized.  It  is  the  responsibility  of  those  in¬ 
volved  in  the  evaluation  process  to  ensure 
that  the  proper  focus  is  maintained  on  key 
issues,  the  T&E  program  yields  informa¬ 
tion  on  critical  technical  and  operational 
issues,  all  data  sources  necessary  for  a 
thorough  evaluation  are  tapped  and  evalu¬ 
ation  results  are  communicated  in  an  ef¬ 
fective  and  timely  manner.  The  evaluation 
process  should  be  evolutionary  through¬ 
out  the  acquisition  phases. 
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Table  14-1.  Sample  Evaluation  Plan 
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MODELING  AND  SIMULATION 
SUPPORT  TO  T&E 


15.1  INTRODUCTION 

This  chapter  discusses  the  applications  of 
modeling  and  simulation  in  test  and  evalu¬ 
ation  (T&E).  The  need  for  modeling  and 
simulation  has  long  been  recognized,  as 
evidenced  by  this  quotation  from  the  US AF 
Scientific  Advisory  Board  in  June  1965: 

Prediction  of  combat  effectiveness  Cem 
only  be,  and  therefore  must  be,  made 
by  using  the  test  data  in  analytical 
procedures.  This  analysis  usually  in¬ 
volves  some  type  of  model,  simula¬ 
tion,  or  game  (i.e.,  the  tools  of  opera¬ 
tions  or  research  analysis).  It  is  the 
exception  and  rarely,  that  the  'end  re¬ 
sult'  i.e.,  combat  effectiveness,  can  be 
deduced  directly  from  test  measure¬ 
ments. 

In  mandating  T&E  early  in  the  acquisition 
process  (i.e.,  before  Milestone  II),  DODI 
5000.2  encourages  the  use  of  modeling  and 
simulation  as  a  source  of  T&E  data.  For 
instance,  the  Armored  Family  of  Vehicles 
program  used  more  than  60  models,  simu¬ 
lations  and  other  test  data  to  support  sys¬ 
tem  concept  exploration.  The  reliance  on 
modeling  and  simulation  by  this  and  other 
acquisition  programs  provides  the  T&E 
community  with  valuable  information 
which  can  increase  confidence  levels,  de¬ 
crease  field  test  time  and  costs,  and  provide 
data  for  pretest  prediction  and  post-test 
validation.  The  Defense  Modeling  and 


Simulation  Office  (DMSO),  working  for  the 
Director  Defense  Research  and  Engineer¬ 
ing,  is  developing  Office  of  the  Secretary  of 
Defense  (OSD)  guidance  on  the  application 
of  modeling  and  simulation  to  the  acquisi¬ 
tion  process. 

This  chapter  discusses  using  modeling  and 
simulation  to  increase  the  efficiency  of  the 
T&E  process,  reduce  time  and  cost,  provide 
otherwise  unattainable  and  immeasurable 
data,  and  provide  more  timely  and  valid 
results. 

15.2  TYPES  OF  MODELS 
AND  SIMULATIONS 

The  term  "modeling  and  simulation"  is 
often  associated  with  huge  digital  com¬ 
puter  simulations;  but  it  also  includes 
manual  and  man-in-the-loop  war  games, 
test  beds,  hybrid  laboratory  simulators  and 
prototypes. 

A  mathematical  model  is  an  abstract  repre¬ 
sentation  of  a  system  that  provides  a  means 
of  developing  quantitative  performance 
requirements  from  which  candidate  de¬ 
signs  can  be  developed.  Static  models  are 
those  that  depict  conditions  of  state  while 
dynamic  models  depict  conditions  that  vary 
with  time,  such  as  the  action  of  an  autopilot 
in  controlling  an  aircraft.  Simple  d)mamic 
models  can  be  solved  analytically,  and  the 
results  represented  graphically. 
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According  to  a  former  Director,  Defense 
Test  and  Evaluation  (Reference  121),  simu¬ 
lations  used  in  T&E  can  be  divided  into 
three  categories: 

...computer  simulations,  system  test 
beds,  and  system  prototypes.  Com¬ 
puter  simulations  are  strictly  math¬ 
ematical  representations  of  systems 
and  do  not  employ  any  actual  hard¬ 
ware.  They  may,  however,  incorpo¬ 
rate  some  of  the  actual  software  that 
might  be  used  in  a  system.  Early  in  a 
system's  life  cycle,  computer  simula¬ 
tions  can  be  expected  to  provide  the 
most  system  evaluation  information. 

In  many  cases,  computer  simulations 
can  be  readily  developed  as  modifica¬ 
tions  of  existing  simulations  for  simi¬ 
lar  systems.  For  example,  successive 
generations  of  AIM-7  missile  simula¬ 
tions  have  been  effectively  used  in  test 
and  evaluation. 

A  system  test  bed  usually  differs  from  a 
computer  simulation  as  it  contains  some, 
but  not  necessarily  all,  of  the  actual  hard¬ 
ware  that  will  be  a  part  of  the  system.  Other 
elements  of  the  system  are  either  not  incor¬ 
porated  or,  if  they  are  incorporated,  are  in 
the  form  of  computer  simulations.  The  sys¬ 
tem  operating  environment  (including 
threat)  may  either  be  physically  simulated, 
as  in  the  case  of  a  flying  test  bed,  or  com¬ 
puter  simulated,  as  in  the  case  of  a  labora¬ 
tory  test  bed.  Aircraft  cockpit  simulators 
used  to  evaluate  pilot  performance  are  good 
examples  of  system  test  beds.  As  develop¬ 
ment  of  a  system  progresses,  more  sub¬ 
systems  become  available  in  hardware 
form.  These  subsystems  can  be  incorpo¬ 
rated  into  system  test  beds  that  typically 
provide  a  great  deal  of  the  system  evalua¬ 
tion  information  used  during  the  middle 
part  of  a  system's  development  cycle. 

The  third  type  of  simulation  used  in  T&E  is 
the  system  prototype.  Unlike  the  system 


test  bed,  all  subsystems  are  physically  in¬ 
corporated  in  a  system  prototype.  The  sys¬ 
tem  prototype  may  closely  represent  the 
final  system  configuration,  depending  on 
the  state  of  development  of  the  various 
subsystems  that  compose  it.  Preproduction 
prototype  missiles  and  aircraft  used  in  op¬ 
erational  testing  by  the  Services  are  ex¬ 
amples  cf  this  class  of  simulation.  As  sys¬ 
tem  development  proceeds,  eventually  all 
subsystems  will  become  available  for  in¬ 
corporation  in  one  or  more  system  proto¬ 
types.  Hard ware-in-the  loop  (HWIL)  simu¬ 
lators  or  full-up  system  simulators  may 
provide  the  foundation  for  continuous  sys¬ 
tem  testing  and  improvement.  These  simu¬ 
lators  can  provide  the  basis  for  transitioning 
hardware  and  software  into  operationally 
realistic  training  devices  with  mission  re¬ 
hearsal  capabilities.  Operational  testing  of 
these  prototypes  frequently  provides  much 
of  the  system  evaluation  information 
needed  for  a  decision  on  full-scale  nroduc- 

A 

tion  and  deployment. 

As  illustrated  in  Figure  15-1,  there  is  a 
continuous  spectrum  of  simulation  types 
with  the  pure  computer  simulation  at  one 
end  and  the  pure  hardware  prototype  at 
the  other  end. 

15.3  VALIDITY  OF  MODELING 
AND  SIMULATION 

Simulations  are  not  a  substitute  for  live 
testing.  There  are  many  things  that  cannot 
be  adequately  simulated  by  computer  pro¬ 
grams;  among  them  are  the  process  of  deci¬ 
sion  and  the  proficiency  of  personnel  in  the 
performance  of  their  functions.  Therefore, 
models  and  simulations  are  not  a  total 
substitution  for  physical  tests  and  evalua¬ 
tions.  Simulations,  manual  and  computer- 
designed,  can  complement  and  increase 
the  validity  of  live  tests  and  evaluations  by 
proper  selection  and  application.  Figure 
15-2  contrasts  the  test  criteria  that  are  con- 
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Figure  15-1.  The  Simulation  Spectrum 


ducive  to  modeling  and  simulation  vs. 
physical  testing.  Careful  selection  of  the 
simulation,  knowledge  of  its  application 
and  operation  and  meticulous  selection  of 
input  data  will  produce  representative  and 
valid  results. 

The  important  element  in  using  a  simula¬ 
tion  is  to  select  one  that  is  representative 
and  either  addresses,  or  is  capable  of  being 
modified  to  address,  the  level  of  detail  (is¬ 
sues,  thresholds  and  objectives)  under  in¬ 
vestigation. 

15.4  SUPPORT  TO  TEST  DESIGN 
AND  PLANNING 

Modeling  and  simulation  can  assist  in  the 
T&E  planning  process  and  can  reduce  the 
cost  of  testing.  Areas  of  particular  applica¬ 
tion  include  scenario  development  and  the 
timing  of  test  events;  the  development  of 
objectives,  essential  elements  of  analysis, 
and  measures  of  effectiveness;  the  identifi¬ 
cation  of  variables  for  control  and  mea¬ 
surement;  and  the  development  of  data 
collection,  instrumentation  and  data  analy¬ 
sis  plans.  For  example,  using  simulation, 
the  test  designer  can  examine  system  sensi¬ 
tivities  to  changes  in  variables  to  determine 
the  critical  variables  and  their  ranges  of 
values  to  be  tested.  He  can  also  predict  the 
effects  of  various  assumptions  and  con¬ 
straints  and  evaluate  candidate  measures 
of  effectiveness  to  help  formulate  the  test 
design. 

Caution  must  be  exercised  when  planning 
to  rely  on  simulations  to  obtain  test  data  as 
they  tend  to  be  expensive  to  develop  or 
modify,  difficult  to  integrate  with  data  from 
other  sources,  and  often  do  not  provide  the 
level  of  realism  required  for  operational 
tests.  Although  simulations  are  not  a  "cure- 
all,"  they  should  be  used  whenever  feasible 
as  another  source  of  data  for  the  evaluator 
to  consider  during  the  test  evaluation. 


Computer  simulations  may  be  used  to  test 
the  planning  for  an  exercise.  By  setting  up 
and  running  the  test  exercise  in  a  simula¬ 
tion,  the  timing  and  scenario  may  be  tested 
and  validated.  Critical  events  may  include 
interaction  of  various  forces  that  test  the 
measures  of  effectiveness  and,  in  turn,  test 
objectives.  Further,  the  simulation  may  be 
used  to  verify  the  statistical  test  design  and 
the  instrumentation,  data  collection,  and 
data  analysis  plans.  Essentially,  the  pur¬ 
pose  of  computer  simulation  in  pretest  plan¬ 
ning  is  to  preview  the  test  to  evaluate  ways 
to  make  test  results  more  effective.  Pretest¬ 
ing  attempts  to  optimize  test  results  by 
pointing  out  potential  trouble  spots.  It  con¬ 
stitutes  a  test  setup  analysis,  which  can 
encompass  a  multitude  of  areas. 

As  an  example  of  simulations  used  in  test 
planning,  consider  a  model  that  portrays 
aircraft  vs.  air  defenses.  The  model  can  be 
used  to  replicate  typical  scenarios  and  pro¬ 
vide  data  on  the  number  of  engagements, 
air  defense  systems  involved,  aircraft  tar¬ 
get,  length  and  quality  of  the  engagement, 
and  a  rough  approximation  of  the  success 
of  the  mission  (i.e.,  if  the  aircraft  made  it  to 
the  target).  With  such  data  available,  a  data 
collection  plan  can  be  developed  to  specify, 
in  more  detail,  when  and  where  data  should 
be  collected,  from  which  systems,  and  in 
what  quantity.  The  results  of  this  analysis 
impact  heavily  on  long  lead-time  items 
such  as  data  collection  devices  and  data 
processing  systems.  The  more  specificity 
available,  the  fewer  the  number  of  sur¬ 
prises  that  will  occur  downstream.  As  tac¬ 
tics  are  decided  upon  and  typical  flight 
paths  are  generated  for  the  scenario,  an 
analysis  can  be  prepared  on  the  flight  paths 
over  the  terrain  in  question;  and  a  determi¬ 
nation  can  be  made  regarding  whether  the 
existing  instrumentation  can  track  the  num¬ 
bers  of  aircraft  involved  in  their  maneuver¬ 
ing  envelopes.  Alternative  site  arrange¬ 
ments  can  be  examined  and  trade-offs  can 
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THE  SIMULATION  SPECTRUM 


Figure  15-2.  Values  of  Selected  Criteria  Conducive  to  Modeling  and  Simulation 


be  made  between  the  amount  of  equipment 
to  be  purchased  and  the  types  of  profiles 
that  can  be  tracked  for  this  particular  test. 
Use  of  such  a  model  can  also  highlight 
numerous  choices  available  to  the  threat  air 
defense  system  in  terms  of  opportimities 
for  engagement  and  practical  applications 
of  doctrine  to  the  specific  situations. 

15.5  SUPPORT  TO  TEST  EXECUTION 

Simulations  can  be  useful  in  test  execution 
and  dynamic  plarming.  With  funds  and 
other  restrictions  limiting  the  number  of 
times  that  a  test  may  be  repeated  and  each 
test  conducted  over  several  days,  it  is  man¬ 
datory  that  the  test  director  exercises  close 
control  over  the  conduct  of  the  test  to  en¬ 
sure  the  specific  types  and  quantities  of 
data  needed  to  meet  the  test  objectives  are 
being  gathered  and  to  ensure  adequate 
safety.  He  must  be  able  to  make  minor 
modifications  to  the  test  plan  and  scenario 
to  force  achievement  of  these  goals.  This 
calls  for  a  dynamic  (quick-look)  analysis 
capability  and  a  dynamic  planning  capa¬ 
bility.  Simulations  may  contribute  to  this 
capability.  For  example,  using  the  same 
simulation(s)  as  used  in  pretest  planning, 
the  tester  could  input  data  gathered  during 
the  first  day  of  the  exercise  to  determine  the 
adequacy  of  the  data  to  fulfill  the  test  objec¬ 
tives.  Using  this  data,  the  entire  test  could 
be  simulated.  Projected  inadequacies  could 
be  isolated,  and  the  test  plans  could  be 
modified  to  minimize  the  deficiencies. 

Simulations  may  also  be  used  to  support 
test  control  and  to  ensure  safety.  For  ex¬ 
ample,  during  missile  test  firings  at  White 
Sands  Missile  Range  ( WSMR),  aerodynamic 
simulations  of  the  proposed  test  were  run 
on  a  computer  during  actual  firings  so  that 
real-time  missile  position  data  could  be 
compared  continuously  to  the  simulated 
missile  position  data.  If  any  significant 


variations  occurred  and  if  the  range  safety 
officer  was  too  slow  (both  types  of  position 
data  were  displayed  on  plotting  boards), 
the  computer  issued  a  destruct  command. 

Simulations  can  be  used  to  augment  tests 
by  simulating  nontestable  events  and  sce¬ 
narios.  Although  operational  testing  should 
be  accomplished  in  as  realistic  an  opera¬ 
tional  environment  as  possible,  pragmati¬ 
cally  some  environments  are  impossible 
to  simulate  for  safety  or  other  reasons.  Some 
of  these  include  the  environment  of  a 
nuclear  battlefield,  to  include  the  effects 
of  nuclear  bursts  on  friendly  and  enemy 
elements.  Others  include  two-sided  live 
firings  and  adequate  representation  of  other 
forces  to  ascertain  compatibility  and 
interoperability  data.  Instrumentation,  data 
collection  and  data  reduction  of  large  com¬ 
bined  armed  forces  (e.g.,  brigade,  division 
and  larger-sized  forces)  become  extremely 
difficult  and  costly.  Simulations  are  not 
restricted  by  safety  factors  and  can  real¬ 
istically  replicate  many  environments  that 
are  otherwise  unachievable  in  an  opera¬ 
tional  test  and  evaluation  (OT &E) — nuclear 
effects,  large  combined  forces,  electronic 
countermeasures  (ECM),  electronic 
counter-countermeasures  (ECCM)  and 
many  engagements. 

Usually,  insufficient  units  are  available  to 
simulate  the  organizational  relationships 
and  interaction  of  the  equipment  with  its 
operational  environment,  particularly  dur¬ 
ing  the  early  OT&E  conducted  using  proto¬ 
type  or  pilot  production-type  equipment. 
Simulations  are  not  constrained  by  these 
limitations.  Data  obtained  from  a  limited 
test  can  be  plugged  into  a  simulation  that  is 
capable  of  handling  many  of  the  types  of 
equipment  being  tested.  It  can  interface 
them  with  other  elements  of  the  blue  forces 
and  operate  them  against  large  elements  of 
the  red  forces  to  obtain  interactions. 
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End-item  simulators  can  be  used  to  evalu¬ 
ate  design  characteristics  of  equipment  and 
can  be  used  to  augment  the  results  ob¬ 
tained  using  prototype  or  pilot  produc¬ 
tion-type  equipment  that  is  representative 
of  the  final  item.  The  simulator  may  be 
used  to  expand  test  data  to  obtain  the  re¬ 
quired  iterations  or  to  indicate  that  the 
human  interface  with  the  prototype  equip¬ 
ment  will  not  satisfy  the  design  require¬ 
ments. 

It  is  often  necessary  to  use  substitute  or 
surrogate  equipment  in  testing;  e.g.,  Ameri¬ 
can  equipment  is  used  to  represent  threat- 
force  equipment.  In  some  cases  the  substi¬ 
tute  equipment  may  have  greater  capabili¬ 
ties  than  the  real  equipment,  in  other  cases 
it  may  have  less.  Simulations  are  capable  of 
representing  the  real  characteristics  of 
equipment  and,  therefore,  can  be  used  as  a 
means  of  modifying  raw  data  collected 
during  the  test  to  reflect  real  characteris¬ 
tics. 

As  an  example,  i.^  the  substitute  equipment 
is  an  AAA  gun  with  a  tracking  rate  of  30 
degrees  per  second  and  the  equipment  for 
which  it  is  substituted  has  a  tracking  rate  of 
45  degrees  per  second,  the  computer  simu¬ 
lation  could  be  used  to  augment  the  col¬ 
lected,  measured  data  by  determining  how 
many  rounds  could  have  been  fired  against 
each  target  or  whether  targets  that  were 
missed  because  the  tracking  rate  was  too 
slow  could  have  been  engaged  by  the  ac¬ 
tual  equipment.  Consideration  of  other  dif¬ 
fering  factors  simultaneously  could  have  a 
plus  or  minus  synergistic  effect  on  test  re¬ 
sults. 

15.6  SUPPORT  TO  ANALYSIS 
AND  TEST  REPORTING 

Modeling  and  simulation  may  be  used  in 
post-test  analysis  to  extend  and  generalize 


results  and  to  extrapolate  to  other  condi¬ 
tions.  The  difficulty  of  instrumenting  and 
controlling  large  exercises  and  collecting 
and  reducing  the  data  and  resource  costs, 
to  some  degree,  limits  the  size  of  T&E.  This 
makes  the  process  of  determining  the  suit¬ 
ability  of  equipment  to  include  compatibil¬ 
ity,  interoperability,  organization,  etc.,  a 
difficult  one.  To  a  large  degree  the  limited 
interactions,  interrelationships  and  com¬ 
patibility  of  large  forces  may  be  supple¬ 
mented  by  using  actual  data  collected  dur¬ 
ing  the  test  and  applying  it  in  the  simula¬ 
tion. 

Simulations  can  be  used  to  extend  test  re¬ 
sults,  save  considerable  energy  (fuel  and 
manpower),  and  save  money  by  reducing 
the  need  to  repeat  data  points  to  improve 
the  statistical  sample  or  to  determine  over¬ 
looked  or  directly  unmeasured  parameters. 
Sensitivity  analyses  can  be  run  using  simu¬ 
lations  to  evaluate  the  robustness  of  the 
design. 

In  analyzing  the  test  results,  data  can  be 
compared  to  the  results  predicted  by  the 
simulations  used  early  in  the  planning  pro¬ 
cess.  Thus,  the  simulation  is  validated  by 
the  actual  live  test  results,  but  the  test  re¬ 
sults  are  also  validated  by  the  simulation. 

15.7  SUMMARY 

Modeling  and  simulation  in  T&E  can  be 
used  for  concept  evaluation,  extrapolation, 
isolation  of  design  effects,  efficiency,  repre¬ 
sentation  of  complex  environments,  and 
overcoming  inherent  limitations  in  actual 
testing.  The  use  of  modeling  and  simula¬ 
tion  can  validate  test  results,  increase  con¬ 
fidence  levels,  reduce  test  costs  and  pro¬ 
vide  opportunities  to  shorten  the  overall 
acquisition  cycle  by  providing  more  data 
earlier  for  the  decision-maker. 


15-7 


16 

TEST  RESOURCES 


16.1  INTRODUCTION 

This  chapter  describes  the  various  types  of 
resources  available  for  testing,  explains 
test  resource  planning  in  the  Services,  and 
discusses  the  ways  in  which  test  resources 
are  funded. 

According  to  DOD 5000.2-M,  the  term  "test 
resources"  is  a  collective  term  that  encom¬ 
passes  elements  necessary  to  plan,  con¬ 
duct,  collect  and  analyze  data  from  a  test 
event  or  program.  These  elements  include: 
funding  (to  develop  new  resources  or  use 
existing  ones),  manpower  for  test  conduct 
and  support,  test  articles,  models,  simula¬ 
tions,  threat  simulators,  surrogates,  repli¬ 
cas,  test-beds,  special  instrumentation,  test 
sites,  targets,  tracking  and  data  acquisition 
instrumentation,  equipment  (for  data  re¬ 
duction,  communications,  meteorology, 
utilities,  photography,  calibration,  secu¬ 
rity,  recovery,  maintenance  and  repair), 
frequency  management  and  control,  and 
base/facility  support  services.  "Testing 
shall  be  planned  and  conducted  to  take  full 
advantage  of  existing  investment  in  EXDD 
ranges,  facilities,  and  other  resources, 
whenever  practical,  unless  otherwise  justi¬ 
fied  in  the  Test  and  Evaluation  Master 
Plan,"  (part  8,  DODI  5000.2). 

Key  DOD  test  resources  are  in  great  de¬ 
mand  by  competing  acquisition  programs. 
Often  special,  unique  or  one-of-a-kind  test 
resources  must  be  developed  specifically 
for  the  test  program.  It  is  imperative  that 


the  requirements  for  these  test  resources  be 
identified  early  in  the  acquisition  process 
so  adequate  funding  can  be  allotted  for 
their  development,  and  they  will  be  avail¬ 
able  when  the  test  is  scheduled. 

16.2  OBTAINING  TEST  RESOURCES 

16.2.1  Major  Range  and  Test  Facility  Base 

All  Services  operate  ranges  and  test  facili¬ 
ties  for  test,  evaluation  and  training  pur¬ 
poses.  Twenty-one  of  these  activities  con¬ 
stitute  the  DOD  Major  Range  and  Test  Fa¬ 
cility  Base  (MRTFB).  This  MRTFB  is  de¬ 
scribed  as  "a  national  asset  which  shall  be 
sized,  operated,  and  maintained  primarily 
for  DOD  test  and  evaluation  support  mis¬ 
sions,  but  also  is  available  to  all  users  hav¬ 
ing  a  valid  requirement  for  its  capabilities. 
The  MRTFB  consists  of  a  broad  base  of  T &E 
(test  and  evaluation]  activities  managed 
and  operated  under  uniform  guidelines  to 
provide  T&E  support  to  DOD  Components 
responsible  for  developing  or  operating 
materiel  and  weapon  systems,"  (Reference 
21).  The  list  of  MRTFB  activities  and  their 
locations  are  shown  on  Figure  16-1.  Sum¬ 
maries  of  the  capabilities  of  each  of  these 
activities  (with  points  of  contact  listed  for 
further  information)  may  be  found  in  DOD 
3200.11-D. 

The  MRTFB  facilities  are  available  for  use 
by  all  the  Services,  other  U.S.  government 
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Figure  16-1 .  DOD  Major  Range  and  Test  Facility  Base 


agencies  and,  in  certain  cases,  allied  foreign 
governments  and  contractor  organizations. 
Scheduling  is  based  on  a  priority  system; 
and  costs  for  usage  are  billed  uniformly,  as 
stated  in  DODD  3200.11.  The  Deputy  Di¬ 
rector,  Test  Facilities  and  Resources  (DTE), 
sets  policy  for  the  composition,  use  and  test 
program  assigiunents  of  the  MRTFB.  In 
turn,  the  individual  Services  must  fund, 
manage  and  operate  their  activities.  They 
are  reimbursed  for  direct  costs  by  each  user 
of  the  activity. 

The  DOD  components  wishing  to  use  an 
MRTFB  activity  must  provide  timely  and 
complete  notification  of  their  requirements, 
such  as  special  instrumentation  or  ground- 
support  equipment  requirements,  to  the 
particular  activity  using  the  documenta¬ 
tion  formats  prescribed  by  Document  501- 
84,  Universal  Docwnentctwn  System  Hand¬ 
book,  issued  by  the  Range  Commanders 
Council.  The  requirements  must  be  stated 
in  the  Test  and  Evaluation  Master  Plan 
(TEMP)  discussed  below.  Personnel  at  the 
MRTFB  activity  will  coordinate  with  and 
assist  prospective  users  with  their  T&E  plan¬ 
ning,  to  include  conducting  trade-off  analy¬ 
ses  and  test  scenario  optimization  based  on 
test  objectives  and  test  support  capabilities. 

16.2.2  Project  Reliance 

In  response  to  a  stated  need  to  consolidate 
DOD  activities  (Defense  Management  Re¬ 
view  Directive  922),  the  Office  of  the  Secre¬ 
tary  of  Defense  (OSD)  T&E  organizations 
have  initiated  a  process  to  review  and  cen¬ 
tralize  various  types  of  system  testing  in¬ 
frastructures  at  designated  Service  test  fa¬ 
cilities.  Project  Reliance  is  focused  on  more 
economical  operations,  allocating  scarce 
funds  for  modernization  and  eliminating 
unwarranted  duplication.  The  Defense  Test 
and  Evaluation  Steering  Group  (DTESG) 
provides  oversight  guidance  and  approval 
of  the  Joint  Commanders  Group  (T&E) 
(JCG(T&E))  recommendations.  Various 


technically  oriented  panels  (Multi-Service 
Test  Investment  Resource  Committee 
(MSTIRC))  review  Service  test  methodol¬ 
ogy  areas  and  develop  supporting  studies 
to  identify  the  Service  test  facility  to  host  a 
particular  type  of  system  test  activity  (i.e., 
anechoic  chambers,  gun  munitions  testing, 
air  breathing  engines,  etc.).  The  MSTIRC 
recommendations  are  reviewed  by  the 
JCG(T&E)  and  forwarded  to  the  DTESG  for 
final  approval.  This  means  all  Services  will 
be  expected  to  use  the  designated  test  facil¬ 
ity  for  that  type  of  testing.  Test  planners 
must  consider  Project  Reliance  agreements 
when  identifying  future  test  sites;  this  will 
necessitate  more  cross-Service  test  support. 

16.2.3  Service  Test  Facilities 

There  are  other  test  resources  available  be¬ 
sides  MRTFB.  The  tester  can  determine 
resources  available  by  contacting  his/her 
Service  headquarters  staff  element  or  if 
within  the  Army,  by  consulting  documents 
such  as  the  Army  Test  and  Evaluation  Com¬ 
mand  (TECOM)  Test  Facilities  Register, 
the  Operational  Test  and  Evaluation  Com¬ 
mand  (OPTEC)  Operational  Test  Instru¬ 
mentation  Guide  and  other  Army  test 
agency  and  range  documents.  Information 
on  specific  Navy  test  resources  is  found  in 
user  manuals  published  by  each  range  and 
the  Commander  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR)  cata¬ 
log  of  available  support. 

16.3  TEST  RESOURCE  PLANNING 

The  development  of  special  test  resources 
to  support  a  weapon  system  test  can  be 
costly  and  time-consuming.  This,  coupled 
with  the  competition  for  existing  test  re¬ 
sources  and  facilities,  requires  that  early 
planning  be  accomplished  to  determine  all 
test  resource  requirements  for  weapon  sys¬ 
tem  T&E.  The  tester  must  use  government 
facilities  whenever  possible  instead  of  fund- 
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ing  construction  of  contractor  test  capabili¬ 
ties. 

Problems  associated  with  range  and  facil¬ 
ity  planning  are  that  major  systems  tend  to 
get  top  priority;  i.e.,  B-IB,  M-1,  etc.  Range 
schedules  are  often  in  conflict  due  to  sys¬ 
tem  problems,  which  cause  schedule  de¬ 
lays  during  testing;  and  there  is  often  a 
shortage  of  funds  to  complete  testing. 

16.3.1  TEMP  Requirements 

The  program  manager  must  state  all  key 
test  resource  requirements  in  the  TEMP 
and  must  include  items  such  as  unique 
instrumentation,  threat  simulators,  surro¬ 
gates,  targets  and  test  articles.  Included  in 
the  TEMP  are  a  critical  analysis  of  antici¬ 
pated  resource  shortfalls,  their  effect  on 
system  T&E  and  plans  to  correct  resource 
deficiencies.  As  the  preliminary  TEMP  must 
be  prepared  for  Milestone  I,  initial  test  re¬ 
source  planning  must  be  accomplished 
during  the  Concept  Exploration  and  Defi¬ 
nition  Phase.  Refinements  and  reassess¬ 
ments  of  test  resource  requirements  are 
included  in  each  TEMP  update.  The  re¬ 
quired  content  of  the  test  resource  sum¬ 
mary  section  of  the  TEMP  is  in  Part  V  -  Test 
and  Evaluation  Resource  Summary,  DOD 
5000.2-M. 

16.3.2  Service  Test  Resource  Planning 

More-detailed  listings  of  required  test  re¬ 
sources  are  generated  in  conjunction  with 
the  detailed  test  plans  written  by  the  mate¬ 
riel  developer  and  operational  tester.  These 
test  plans  describe  test  objectives,  measures 
of  effectiveness  (MOEs),  scenarios  and  spe¬ 
cific  test  resource  requirements. 

16.3.2.1  Army  Test  Resource  Planning 

In  the  Army,  the  tester  prepares  input  to 
the  TEMP  and  the  Test  and  Evaluation  Plan 


(TEP),  the  primary  planning  documents 
for  operational  test  and  evaluation  (OT&E) 
of  the  weapon  system.  These  documents 
should  be  prepared  early  in  the  acquisition 
cycle  (at  the  beginning  of  the  Concept  Dem¬ 
onstration  and  Validation  Phase).  They 
describe  the  entire  T&E  approach  includ¬ 
ing  critical  issues,  test  methodology,  mea¬ 
sures  of  effectiveness  and  all  necessary  test 
resources.  The  TEMP  and  TEP  provide  the 
primary  input  to  the  Outline  Test  Plan 
(OTP),  which  contains  a  detailed  descrip¬ 
tion  of  each  identified  required  test  resource, 
where  and  when  it  is  to  be  provided,  and 
the  providing  organization. 

The  tester  must  coordinate  the  OTP  with  all 
major  commands  or  agencies  expected  to 
provide  test  resources.  Then,  the  OTP  is 
submitted  to  the  Resource  Management 
Division,  HQ,  OPTEC,  for  review  by  the 
Test  Schedule  and  Review  Committee 
(TSARC)  and  for  incorporation  into  the 
Army's  Five-Year  Test  Program  (FYTP). 
The  initial  OTP  for  each  test  should  be 
submitted  to  the  TSARC  as  soon  as  testing 
is  identified  in  the  TEMP.  Revised  OTPs  are 
submitted  as  more  information  becomes 
available  or  requirements  change,  but  a 
final  comprehensive  version  of  the  OTP 
should  be  submitted  at  least  18  months 
before  the  resources  are  required. 

The  TSARC  is  responsible  for  providing 
high-level,  centralized  management  of  T&E 
resource  planning.  The  TSARC  is  chaired 
by  the  Commanding  General  OPTEC  and 
consists  of  a  general  officer  or  equivalent 
representatives  from  the  Army  staff  and 
major  commands.  The  TSARC  meets  semi¬ 
annually  to  review  all  OTPs,  resolve  con¬ 
flicts  and  coordinate  all  identified  test  re¬ 
source  requirements  for  inclusion  in  the 
FYTP.  The  FYTP  is  a  formal  resource  task¬ 
ing  document  for  current  and  near-term 
tests  and  a  planning  document  for  tests 
scheduled  for  the  out-years.  All  OTPs  are 
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reviewed  during  the  semiannual  reviews 
to  ensure  that  any  refinements  or  revisions 
are  approved  by  the  TSARC  and  reflected 
in  the  FYTP.  The  FYTP  is  produced  as  a 
hard-copy  by  OPTEC. 

The  TSARC-approved  OTP  is  a  tasking 
document  by  which  the  tester  requests 
Army  test  resources.  The  TSARC  coordi¬ 
nates  resource  requests,  sets  priorities,  re¬ 
solves  conflicts  and  schedules  resources. 
The  resultant  FYTP,  when  approved  by  the 
Deputy  Chief  of  Staff  for  Operations  and 
Plans  (DCSOPS),  HQ  DA,  is  a  formal  task¬ 
ing  document  that  reflects  the  agreements 
made  by  the  resource  providers  (Army 
Materiel  Command  (AMC),  Training  and 
Doctrine  Command  (TRADOC),  Forces 
Command  (FORSCOM),  etc.)  to  make  the 
required  test  resources  available  to  the  des¬ 
ignated  tests.  If  test  resources  from  another 
Service,  a  non-DCD  governmental  agency 
(such  as  the  Department  of  Energy  (EXDE) 
or  NASA)  or  a  contractor  are  required,  the 
request  is  coordinated  by  the  OPTEC  Re¬ 
source  Management  Division.  For  example, 
the  request  for  a  range  must  be  made  at 
least  two  years  in  advance  to  ensure  avail¬ 
ability.  However,  due  to  the  long  lead  time 
required  to  schedule  these  non- Army  re¬ 
sources,  their  availability  caimot  be  guar¬ 
anteed  if  testing  is  delayed  or  retesting  is 
required.  The  use  of  resources  outside  the 
U.S.,  such  as  in  Canada,  Germany  or  other 
NATO  countries,  is  also  handled  by  OPTEC. 

16.3.2.2  Navy  Test  Resource  Planning 

In  the  Navy,  the  developing  agency  and  the 
operational  tester  are  responsible  for  iden¬ 
tifying  the  specific  test  resources  required 
in  testing  the  weapon  system.  In  develop¬ 
ing  requirements  for  test  resources,  the  pro¬ 
gram  manager  (PM)  and  operational  test 
director  (OTD)  refer  to  documents  such  as 
the  Mission  Need  Statement  (MNS),  Inte¬ 
grated  Program  Summary  (IPS),  Navy  Deci¬ 


sion  Coordinating  Paper  (NEXTP),  Opera¬ 
tional  Requirement  Document  (ORD), 
threat  assessments.  Office  of  the  Chief 
of  Naval  Operations  Instruction 
(OPNAVINST)  5000.42D  (Test  and  Evalua¬ 
tion),  and  the  OTD  Guide  (Commander, 
Operation  Test  and  Evaluation  Force  (Navy) 
(COMOPTEVFOR)  Instruction  3960.1D). 
Upon  Chief  of  Naval  Operations  (CNO) 
approval,  the  TEMP  becomes  the  control¬ 
ling  management  document  for  all  T&E  of 
the  weapon  system.  It  constitutes  direction 
by  the  CNO  to  conduct  the  T&E  program 
defined  in  the  TEMP,  including  the  com¬ 
mitment  of  research,  development,  test  and 
evaluation  (RDT&E)  financial  support  and 
of  fleet  units  and  schedules.  It  is  prepared 
by  the  PM,  who  is  provided  OT&E  input  by 
the  COMOPTEVFOR  Operational  Test  Di¬ 
rector.  The  TEMP  defines  all  T&E  (devel¬ 
opment  test  and  evaluation  (DT&E),  OT&E 
and  production  acceptance  test  and  evalu¬ 
ation  (PAT&E))  to  be  conducted  for  the 
system  and  describes,  in  as  much  detail  as 
possible,  the  test  resources  required. 

The  Navy  uses  its  operational  naval  forces 
to  provide  realistic  T&E  of  new  weapon 
systems.  Each  year,  the  CNO  (N-091)  com¬ 
piles  all  Fleet  support  requirements  for 
RDT&E  program  support  from  the  TEMPs 
and  publishes  the  CNO  Long-Range 
RDT&E  Support  Requirements  document 
for  the  budget  and  out-years.  In  addition,  a 
quarterly  forecast  of  support  requirements 
is  published  approximately  five  months 
before  the  Fleet  Employment  Scheduling 
Conference  for  the  quarter  in  which  the 
support  is  required.  These  documents  sum¬ 
marize  OT&E  requirements  for  Fleet  ser¬ 
vices  and  are  used  by  the  Fleet  for  schedul¬ 
ing  services  and  out-year  budget  projec¬ 
tions. 

Requests  for  use  of  range  assets  are  usually 
initiated  informally  with  a  phone  call  from 
the  PM  and/or  OTD  to  the  range  manager 
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and  followed  by  formal  documentation. 
Requests  for  Fleet  support  are  usually  more 
formal.  The  COMOPTEVFOR,  in  coordi¬ 
nation  with  the  PM,  forwards  the  TEMP 
and  a  Fleet  RDT&E  Support  Request  to  the 
CNO.  Upon  approval  of  the  request,  the 
CNO  tasks  the  Fleet  Commander  in  Chief 
(CINC)  by  letter  or  message  to  coordinate 
with  OPTEVFOR  to  provide  the  requested 
support. 

Use  of  most  Navy  ranges  must  be  sched¬ 
uled  at  least  a  year  in  advance.  Each  range 
consolidates  and  prioritizes  user  requests, 
negotiates  conflicts  and  attempts  to  sched¬ 
ule  range  services  to  satisfy  all  requests.  If 
the  desired  range  services  cannot  be  made 
available  when  required,  the  test  must  wait; 
or  the  CNO  resolves  the  conflict.  Because 
ranges  are  fully  scheduled  in  advance,  it  is 
difficult  to  accommodate  a  test  that  is  de¬ 
layed  or  requires  additional  range  time 
beyond  that  originally  scheduled.  Again, 
the  CNO  can  examine  the  effects  of  delays 
or  retest  requirements  and  issue  revised 
priorities,  as  required. 

Requests  for  use  of  non-Navy  OT&E  re¬ 
sources  are  initiated  by  COMOPTEVFOR. 
The  Operational  Test  and  Evaluation  Force 
(OPTEVFOR)  is  authorized  direct  liaison 
with  other  Service-independent  operational 
test  agencies  (OTAs)  to  obtain  OTA-con- 
trolled  resources.  Requests  for  other  gov¬ 
ernment-owned  resources  are  forwarded 
to  the  CNO  (N-091)  for  formal  submission 
to  the  Service  Chief  (for  Service  assets)  or  to 
the  appropriate  government  agency  (e.g., 
EXDE  or  NASA).  Use  of  contractor  resources 
is  usually  handled  by  the  PM,  although 
contractor  assets  are  seldom  required  in 
OT&E,  since  the  Fleet  is  used  to  provide  an 
operational  environment.  Requests  for  use 
of  foreign  ranges  are  handled  by  the  N-091 
Assistant  for  International  Research  and 
Development  (R&D). 


16.3.2.3  Air  Force  Test  Resource  Planning 

The  test  resources  required  for  T&E  of  an 
Air  Force  weapon  system  are  identified  in 
detail  in  the  Test  Program  Outline  (TPO), 
which  is  prepared  by  the  responsible  Air 
Force  T&E  organization.  In  general,  the  Air 
Force  Operational  Tests  and  Evaluation 
Center  (AFOTEC)  is  the  test  organization 
for  OT&E  programs;  it  obtains  support  from 
a  Service  major  command  test  agency  for 
nonmajor  programs,  with  AFOTEC  direct¬ 
ing  and  providing  assistance,  as  required. 

During  the  Advanced  Planning  Phase  of  a 
weapon  system  acquisition  (five  to  six  years 
before  OT&E),  AFOTEC  prepares  the  OT &E 
section  of  the  first  full  TPO,  coordinates  the 
TPO  with  all  supporting  organizations  and 
assists  the  resource  manager  (RM)  in  pro¬ 
gramming  required  resources.  The  resource 
requirements  listed  in  the  Resource  Infor¬ 
mation  Network  TPO  are  developed  by  the 
test  manager,  resource  manager  and  test 
support  group,  using  sources  such  as  the 
ORD  and  threat  assessments.  The  TPO 
should  specify,  in  detail,  all  the  resources 
necessary  to  successfully  conduct  a  test 
when  it  is  entered  in  the  Test  Resource 
Information  Management  System  (TRIMS). 

The  TPO  is  the  formal  means  by  which  test 
resource  requirements  are  communicated 
to  the  Air  Staff  and  to  the  appropriate  com¬ 
mands  and  agencies  tasked  to  supply  the 
needed  resources.  Hence,  if  a  required  re¬ 
source  is  not  specified  in  the  TPO,  it  is  likely 
the  resource  will  not  be  available  for  the 
test.  The  TPO  is  revised  and  updated  on  a 
continuous  basis,  since  the  test  resource 
requirements  become  better  defined  as  the 
OT&E  plans  mature.  The  initial  TPO  serves 
as  a  baseline  for  comparison  of  planned 
OT&E  resources  with  actual  expenditures. 
Comparisons  of  the  initial  TPO  with  subse¬ 
quent  updates  provide  an  audit  trail  of 
changes  in  the  test  program  and  its  testing 
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requirements.  The  AFOTEC  maintains  all 
TPOs  on  TRIMS;  this  permits  immediate 
response  to  all  queries  regarding  test  re¬ 
source  requirements. 

The  AFOTEC /RM  consolidates  the  re¬ 
source  requirements  from  all  TPOs  coordi¬ 
nating  with  participating  and  supporting 
organizations  and  agencies  outside 
AFOTEC.  Twice  yearly,  the  RM  office  pre¬ 
pares  a  draft  of  the  USAF  Program  for 
Operational  Test  (PO).  The  PO  is  a  master 
planning  and  programming  document  for 
resource  requirements  for  all  HQ  USAF- 
directed  OT&E  and  is  distributed  to  all 
concerned  commands,  agencies  and  orga¬ 
nizations  for  review  and  coordination.  It  is 
then  submitted  to  the  Air  Staff  for  review 
and  approval  by  the  Operational  Resource 
Management  Assessment  System  for  Test 
and  Evaluation  (ORMAS/TE),  which  op¬ 
erates  under  the  authority  of  HQ  AF/TE. 
The  ORMAS  Board  is  composed  of  HQ 
USAF  action  officers  and  senior  officers 
from  major  commands  (MAJCOMs)  and 
agencies  involved  in  OT&E;  it  meets  to 
resolve  impacts  and  conflicting  require¬ 
ments  at  the  appropriate  Air  Staff  level. 
Through  the  ORMAS  process,  HQ  USAF 
approves  the  PO,  which  becomes  a  direc¬ 
tive  to  participants  for  planning,  program¬ 
ming  and  budgeting  actions.  Agreements 
made  among  ORMAS  participants  regard¬ 
ing  TPO  and  PO  resource  requirements  are 
considered  binding. 

All  requests  for  test  resources  are  coordi¬ 
nated  by  HQ  AFOTEC  as  part  of  the  TPO 
preparation  process.  When  a  new  weapon 
system  development  is  first  identified, 
AFOTEC  provides  a  test  manager  (TM) 
who  begins  long-term  OT &E  planning.  The 
TM  begins  identifying  needed  test  re¬ 
sources,  such  as  instrumentation,  simula¬ 
tors  and  models,  and  works  with  the  re¬ 
sources  directorate  to  obtain  them.  If  the 
required  resource  does  not  belong  to 


AFOTEC,  it  will  negotiate  with  the  com¬ 
mands  having  the  resource.  In  the  case  of 
models  and  simulators,  AFOTEC  surveys 
what  is  available,  assesses  credibility,  and 
then  coordinates  with  the  owner  or  devel¬ 
oper  to  use  it.  The  Joint  Technical  Coordi¬ 
nating  Group  publishes  a  document  on 
electronic  warfare  (EW)  models. 

Range  scheduling  should  be  done  early.  At 
least  a  year  is  required,  but  often  a  test  can 
be  accommodated  with  a  few  months'  no¬ 
tice  if  there  is  no  requirement  for  special 
equipment  or  modifications  to  be  provided 
at  the  range.  Some  of  the  Air  Force  ranges 
are  scheduled  well  in  advance  and  cannot 
accommodate  tests  that  encounter  delays 
or  retest  requirements. 

The  resource  manager  attempts  to  resolve 
conflicts  among  various  systems  compet¬ 
ing  for  scarce  test  resources  and  elevates 
the  request  to  the  Commander,  AFOTEC,  if 
necessary.  Decisions  on  resource  utiliza¬ 
tion  and  scheduling  are  based  on  the 
weapon  system's  assigned  priority. 

The  resource  manager  and  the  test  man¬ 
ager  also  arrange  for  use  of  the  resources  of 
other  Services,  non-DOD  government  agen¬ 
cies  and  contractors.  Use  of  non-U.S.  re¬ 
sources,  such  as  a  Canadian  range,  are  co¬ 
ordinated  by  AF/TE  and  based  on  formal 
Memoranda  of  Understanding  (MOU).  The 
USAFE/EXDQ  handles  requests  for  Euro¬ 
pean  ranges.  Use  of  a  contractor-owned 
resource,  such  as  a  model,  is  often  obtained 
through  the  System  Program  Office  (SPO) 
or  a  general  support  contract. 

16.4  TEST  RESOURCE  FUNDING 

The  Future  Years  Defense  Program  (FYDP), 
incorporating  a  biennial  budgeting  pro¬ 
cess,  is  the  basic  DOD  programming  docu¬ 
ment  that  records,  summarizes  and  dis¬ 
plays  Secretary  of  Defense  (SECDEF)  deci- 
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sions.  In  the  FYDP,  costs  are  divided  into 
three  categories  for  each  acquisition  pro¬ 
gram  element:  research  and  development 
costs,  in  vestment  costs  and  operating  costs. 
The  Congress  appropriates  to  the  Office  of 
Management  and  Budget  (OMB),  and  OMB 
apportions  fimding  through  the  SECDEF 
to  the  Services  and  to  other  defense  agen¬ 
cies.  The  Services  and  defense  agencies  then 
allocate  funds  to  others  (claimants, 
subclaimants,  administering  offices,  com¬ 
manding  generals,  etc.). 

The  Plaiming,  Programming,  and  Budget¬ 
ing  System  (PPBS)  is  a  DOD  internal  sys¬ 
tem  used  to  develop  input  to  the  Congress 
for  each  year's  budget  while  developing 
future-year  budgets.  The  PPBS  is  calendar 
oriented.  There  are  concurrent  two-year 
PPBS  cycles  ongoing  at  one  time.  These 
cycles  are:  planning,  programming  and 
budgeting.  At  any  one  time  there  are  three 
budgets  being  worked  by  the  Services.  The 
current  two-year  budget  is  being  executed. 
The  next  six  years  of  defense  planning  is 
being  programmed,  and  long-range  pro¬ 
gram  plans  and  planning  guidance  are  be¬ 
ing  reviewed  for  updating. 

There  are  six  types  of  funding  in  the  PPBS: 
research  funding  for  maintaining  the  tech¬ 
nology  base;  exploratory  development 
funding  for  conducting  the  Concept  Explo¬ 
ration  and  Definition  Phase;  advanced  de¬ 
velopment  funding  for  conducting  both 
the  Concept  Exploration  and  Definition 
Phase  and  the  Demonstration  and  Valida¬ 
tion  Phase;  engineering  development  fund¬ 
ing  for  conducting  the  Engineering  and 
Manufacturing  Development  Phase;  opera¬ 
tional  systems  development  funding  for 
conducting  the  Production  and  Deploy¬ 
ment  Phase;  and  RDT&E  management  and 
support  funding,  which  is  used  through¬ 
out  the  development  and  production  cycle 
imtil  the  system  is  operationally  deployed 
when  operations  and  maintenance  (O&M) 


funding  is  used.  The  RDT&E  appropria¬ 
tion  funds  the  costs  associated  with  re¬ 
search  and  development,  including  test 
items,  DT&E  and  test  support  of  OT&E  of 
the  system  or  equipment  and  the  test  items. 

The  funding  that  is  planned,  programmed 
and  budgeted  through  the  PPBS  cycle  is  not 
always  the  same  fimding  amount  that  the 
Congress  appropriates  or  the  PM  receives. 
If  the  required  funding  for  a  test  program  is 
not  authorized  by  the  Congress,  the  PM  has 
four  ways  to  react.  The  PM  can  submit  a 
supplemental  budget  (for  unfunded  por¬ 
tions  of  the  program),  request  deficiency 
funding  (for  unforeseen  program  problems) 
or  use  transfer  authority  (from  other  pro¬ 
grams  within  the  Service);  or  the  PM  can  try 
to  reprogram  the  needed  funds  (to  restruc¬ 
ture  the  program). 

Generally,  testing  that  is  accomplished  for 
a  specific  system  before  the  production 
decision  is  funded  from  RDT&E  appro¬ 
priations;  and  testing  that  is  accomplished 
after  the  production  decision  is  funded  from 
other  procurement  or  operations  and  main¬ 
tenance  appropriations.  Testing  of  product 
improvements,  block  upgrades  and  major 
modifications  is  funded  from  the  same  ap¬ 
propriations  as  the  program  development. 
Follow-on  Test  and  Evaluations  (FOT&E) 
are  normally  funded  from  O&M  funds. 

Funding  associated  with  T&E  (including 
instrumentation,  targets  and  simulations) 
are  identified  in  the  system  acquisition  cost 
estimates.  Service  acquisition  plans  and 
the  TEMP.  General  funding  information 
for  development  and  operational  tests  fol¬ 
lows: 

Development  Test  Funding.  Funds  required 
to  conduct  engineering  and  development 
tests  are  programmed  and  budgeted  by  the 
materiel  developer,  based  upon  the  require¬ 
ments  of  the  TEMP.  These  costs  may  in- 
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elude,  but  are  not  limited  to,  procuring  test 
samples /prototypes;  support  equipment; 
transportation  costs;  technical  data;  train¬ 
ing  of  test  personnel;  repair  parts;  and  test- 
specific  instrumentation,  equipment  and 
facilities.  The  DT&E  funds  are  expended 
for  contractor  and  government  develop¬ 
mental  test  activities. 

The  Service  PM  may  be  required  to  pay  for 
the  use  of  test  resources,  such  as  the  MRTFB, 
and  for  the  development  of  specialized  re¬ 
sources  needed  specifically  for  testing  the 
weapon  system  being  developed. 

Operational  Test  fOTl  Funding.  Funds  re¬ 
quired  to  conduct  OT  are  usually  pro¬ 
grammed  and  budgeted  by  the  ^rvice 
operational  test  agency  or  organization. 
The  funds  are  programmed  in  the  Service's 
long-range  test  program,  and  the  funds 
requirements  are  obtained  from  the  test 
resourcing  documentation  and  TEMP. 

16.4.1  Army  Funding 

Test  resources  are  developed  and  funded 
under  various  Army  appropriations.  The 
Army  Materiel  Command  and  its  commod¬ 
ity  commands  provide  test  items,  spare 
parts,  support  items  (such  as  diagnostic 
equipment)  and  ammunition.  Soldiers, 
ranges,  fuel,  test  support  personnel  and 
maneuver  areas  are  provided  by  TRADOC 
or  FORSCOM.  The  weapon  system  PM 
uses  RDT&E  funds  to  reimburse  these  sup¬ 
porting  commands  for  costs  directly  re¬ 
lated  to  his  test.  The  weapon  system  mate¬ 
riel  developer  is  also  responsible  for  fund¬ 
ing  the  development  of  new  test  resources 
specifically  needed  to  test  the  weapon  sys¬ 
tem.  Examples  of  such  special-purpose  re¬ 
sources  include  models,  simulations,  spe¬ 
cial  instrumentation  and  test  equipment, 
range  modifications,  EW  simulators  and, 
sometimes,  threat  simulators.  Although  the 
Army  has  a  separate  budget  and  develop¬ 


ment  plan  for  threat  simulators,  the  Army 
Development  and  Acquisition  of  Threat 
Simulators  (ADATS)  program,  many 
weapon  system  developers  still  have  to 
fund  the  cost  of  new  threat  systems  that  are 
specifically  needed  to  test  their  weapon 
system.  Army  OPTEC  is  funded  through 
its  own  program  element  and  has  direct 
control  of  OT&E  funds  for  all  programs. 
Funding  requirements  are  developed  in 
consonance  with  the  Outline  Test  Plan. 

16.4.2  Navy  Funding 

In  the  Navy,  the  weapon  system  PM  is 
responsible  for  funding  the  development 
of  all  required  test-specific  resources  from 
the  program's  RDT&E  funds.  These  re¬ 
sources  include  test  articles,  expendables, 
one-of-a-kind  targets,  data  collection /re¬ 
duction  and  instrumentation.  The  devel¬ 
opment  of  generic  test  resources  that  can  be 
used  in  OT&E  of  multiple  weapon  systems 
such  as  targets,  threat  simulators  and  range 
capabilities,  is  funded  from  OPNAV  ge¬ 
neric  accounts  (such  as  target  development) 
and  not  from  weapon  systems  RDT&E.  The 
PM's  RDT&E  funds  pay  for  all  DT  and  OT 
through  OPEVAL.  The  PM  pays  for  all 
post-production  OT  with  program  funds. 

16.4.3  Air  Force  Funding 

In  the  Ai  •  Force,  direct-cost  funding  re¬ 
quires  that  test-peculiar  (direct)  costs  asso¬ 
ciated  with  a  particular  test  program  be 
reimbursed  by  the  System  Program  Office 
to  the  designated  test  agency.  The  RDT&E 
appropriation  funds  the  cost  associated 
with  research  and  development,  including 
test  items,  DT&E  and  Air  Force  Materiel 
Command  (AFMC)  support  of  OT&E  of 
the  system  or  equipment  and  the  test  items. 
Costs  associated  with  initial  operational 
test  and  evaluation  (lOT&E)  are  RDT&E 
funded,  and  costs  of  qualification  opera¬ 
tional  test  and  evaluation  (QOT&E)  are 
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O&M  funded.  The  AFOTEC  is  funded 
through  its  own  program  element  and  has 
direct  control  of  OT&E  funds  for  all  pro¬ 
grams.  The  lOT&E  manager  prepares  a 
TPO  that  summarizes  the  resource  require¬ 
ments  for  lOT&E  and  related  test  support. 
All  pretest  lOT&E  planning  is  budgeted 
through  and  paid  out  of  the  O&M  appro¬ 
priation.  The  FOT&E  costs  are  paid  by 
AFOTEC  and/or  the  MAJCOM  operating 
the  system  and  funded  by  the  O&M  appro¬ 
priation. 

16.5  SUMMARY 

Test  resources  have  many  conflicting  de¬ 
mands  and  their  use  must  be  scheduled 
well  in  advance  of  a  test.  Resources  specific 
to  a  particular  test  must  often  be  developed 
and  funded  from  the  PM's  own  RDT&E 
budget.  Thus,  the  PM  and  his  testers  must 
ensure  that  test  resource  requirements  are 
identified  early  in  the  acquisition  cycle. 


that  they  are  documented  in  the  initial 
TEMP,  and  that  modifications  and  refine¬ 
ments  are  reported  in  the  TEMP  updates. 

Funds  for  testing  are  provided  by  congres¬ 
sional  appropriation  to  the  OMB,  ’  hich 
apportions  the  funds  to  the  Services  through 
the  SECDEF.  The  PPBS  is  the  DOD  process 
used  to  formulate  budget  requests  to  the 
Congress.  Requests  by  PMs  for  test  re¬ 
sources  ar'^  usually  outlined  in  the  TEMP. 
Generally,  system  development  is  funded 
from  RDT&E  funds  until  the  system  is  op¬ 
erationally  deployed  and  maintained,  and 
O&M  funds  are  used  for  FOT&E  and  sys¬ 
tem  maintenance.  The  weapon  system 
materiel  developer  is  also  responsible  for 
funding  the  development  .jf  new  test  re¬ 
sources  specifically  needed  to  test  the 
weapon  system.  Army  and  Air  Force  op¬ 
erational  test  agencies  develop  and  directly 
control  OT&E  funds  for  their  Services. 
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Table  16-1 .  TEMP  Test  Resource  Summary  Section 


Part  V-'Test  and  Evaluation  Research  Summary:  Provide  a  summary  of  all 
key  resources,  government  and  contractor  planned,  to  be  used  during  the 
acquisition  program.  The  initial  TEMP  should  project  those  key  resources, 
including  major  range  and  unique  instrumentation  requirements,  threat 
simulators  and  targets,  necessary  to  accomplish  DT&E  and  OT&E 
objectives.  As  system  development  progresses,  test  resource  requirements 
shall  be  reassessed,  and  subsequent  TEMP  updates  shall  reflect  any 
changed  system  concepts  or  requirements  and/or  updated  threat 
assessments.  Specifically,  the  TEMP  shall  identify,  as  applicable,  the 
following  test  resources: 


•Test  Articles 

•Test  Sites  and  Instrumentation 

•Test  Support  Equipment 

•Threat  Systems/Simulators 

•Test  Targets  and  Expendables 

•Operational  Force  Test  Support 

•Simulators,  Models  and  Test-Beds 

•Special  Requirements 

•Test  and  Evaluation  Funding  Requirements 

•Manpower/Personnel  Training 


Source:  DOD  5000.2-M 
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17 

TEST  AND  EVALUATION  MASTER  PLAN 


17.1  INTRODUCTION 

Guidance  contained  in  DODI  5000.2  stipu¬ 
lates  "a  Test  and  Evaluation  Master  Plan 
will  be  prepared  for  all  acquisition  pro¬ 
grams."  This  reinforces  the  philosophy  that 
good  plaiming  supports  good  operations. 
For  effective  engineering  development  and 
decision-making  processes,  an  overall  strat¬ 
egy  must  be  developed  integrating  the  col¬ 
lection  and  evaluation  of  test  data  on  re¬ 
quired  performance  parameters.  The  Test 
and  Evaluation  Master  Plan  (TEMP)  "re¬ 
lates  program  schedule,  test  management 
strategy  and  structure,  and  required  re¬ 
sources  to:  critical  operational  issues;  criti¬ 
cal  technical  parameters;  minimum  accept¬ 
able  operational  performance  require¬ 
ments;  evaluation  criteria,  and,  milestone 
decision  points."  Feedback  about  the  de¬ 
gree  of  system  performance  maturity  and 
its  operational  effectiveness  and  suitability 
during  each  phase  is  essential  to  the  suc¬ 
cessful  fielding  of  equipment  that  satisfies 
user  requirements. 

17.2  TEMP  DEVELOPMENT 

The  development  of  program  test  and 
evaluation  (T&E)  strategy,  codification  in 
the  TEMP,  and  effective  management  of 
the  various  test  processes  is  one  of  the 
primary  functions  of  a  program  manage¬ 
ment  office.  The  T&E  strategy  is  highly 
contingent  on  Phase  0  concept(s)  that  are 
deemed  appropriate  for  satisfying  user  re¬ 
quirements.  As  outlined  in  DODD  5000.1, 


part  1,  the  priority  for  selecting  a  solution 
is: 

(1)  a  non-materiel  solution,  such  as 
changes  to  tactics,  doctrine,  operational 
concepts,  training,  or  organization. 

(2)  the  sequence  of  materiel  alterna¬ 
tives  is: 

(a)  use  or  modification  of  an  existing 
US  military  system. 

(b)  use  or  modification  of  an  existing 
commercially  developed  or  Allied  sys¬ 
tem  that  fosters  a  non-developmental 
acquisition  strategy. 

(c)  a  cooperative  research  and  devel¬ 
opment  program  with  one  or  more 
Allied  nations. 

(d)  a  new  joint-Service  development 
program. 

(e)  a  new  service-imique  development 
program. 

The  quality  of  the  test  program  may  di¬ 
rectly  reflect  the  level  of  effort  expended  in 
its  development  and  execution.  This  varies 
in  direct  relationship  to  the  management 
imposed  by  the  program  manager  (PM) 
and,  to  some  extent,  by  the  system  engi¬ 
neer.  The  PM  must  evaluate  the  utility  of 
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dedicated  T&E  staff  vs.  matrix  support  from 
the  development  command.  The  levels  of 
intensity  for  planning  and  executing  T&E 
fluctuate  with  changes  in  phases  of  the 
acquisition  process  and  in  T&E  staff  sup¬ 
port,  as  appropriate. 

Early  planning  of  long-range  strategies  can 
be  supported  with  Imowledgeable  plan¬ 
ning  teams  (Test  Integration  Working 
Groups  (TIWGs),  Test  Planning  Working 
Groups  (TPWGs))  and  reviews  by  panels 
of  senior  T&E  management  officials  — 
"gray  beards."  As  the  tempo  of  actual  test 
activities  begins  to  build  (late  Demonstra¬ 
tion/Validation  Phase  (Dem/Val)  to  pre- 
LRIP  (low-rate  initial  production)  Engineer¬ 
ing  and  Manufacturing  Development 
(EMD)  Phase,  internal  T&E  management 
staff  is  needed  to  control  the  processes  and 
evaluate  results. 

17.2.1  Program  Management  Office 
Responsibilities 

The  Program  Management  Office  (PMO)  is 
the  focal  point  of  the  development,  review 
and  approval  process  for  the  program 
TEMP.  The  DOD  acquisition  process  re¬ 
quires  a  TEMP  as  one  of  the  primary  man¬ 
agement  strategy  documents  supporting 
the  decision  to  start  or  terminate  develop¬ 
ment  efforts  at  Milestone  I.  This  task  is  a 
"difficult  do"  during  the  Concept  and  Defi¬ 
nition  Phase  since  some  Services  do  not 
formulate  or  staff  a  PMO  until  program 
start  (Milestone  (MS)  I).  An  additional  com¬ 
plicating  factor  is  the  nebulous  condition  of 
other  program  source  documents  (Opera¬ 
tional  Requirement  Document  (ORD),  Sys¬ 
tem  Engineering  Management  Plan 
(SEMP),  Acquisition  Strategy,  System 
Threat  Assessment  Report  (STAR),  Inte¬ 
grated  Logistics  Support  Plan  (ILSP),  etc.) 
that  are  also  in  early  stages  of  develop¬ 
ment/  updating  for  the  milestone  review. 
Since  the  TEMP  must  conform  to  other 
program  management  documents,  it  fre¬ 


quently  lags  in  the  development  process 
and  does  not  receive  the  attention  needed 
from  PMO  or  matrix  support  personnel. 
Program  Management  Office  emphasis  on 
early  formulation  of  the  test  planning  teams 
(TIWG,  TPWG)  is  critical  to  the  successful 
development  of  the  program  TEMP.  These 
teams  should  consist  of  the  requisite  play¬ 
ers  so  a  comprehensive  and  integrated  strat¬ 
egy  compatible  with  other  engineering  and 
decision-making  processes  is  developed. 
The  PMO  will  find  that  the  number  of 
parties  desiring  coordination  on  the  TEMP 
far  exceed  the  "streamlined"  approval  pro¬ 
cess  signatories  (part  7,  EXDD  5000.2-M). 
However,  it  must  be  coordinated;  an  early 
start  in  getting  Service-level  concurrence  is 
important  so  the  Defense  Acquisition  Board 
(DAB)  document-submission  schedule  can 
be  supported  with  the  draft  and  final  ver¬ 
sions  of  the  TEMP.  Subsequent  updates  do 
not  become  easier,  as  each  acquisition  phase 
brings  new  planning,  coordination  and  test¬ 
ing  requirements. 

17.2.2  T&E  Planning 

Developing  an  overall  strategy  provides 
the  framework  for  incorporating  phase- 
oriented  T&E  activities  that  will  facilitate 
the  acquisition  process.  The  T&E  strategy 
should  be  consistent  with  the  program  ac¬ 
quisition  strategy,  identifying  requirements 
for  contractor  and  government  develop¬ 
ment  test  and  evaluation  (DT&E),  interac¬ 
tions  between  DT&E  and  operational  test 
and  evaluation  (OT&E),  and  provisions  for 
the  separate  initial  operational  test  and 
evaluation  (lOT&E).  An  evolutionary  ac¬ 
quisition  strategy  will  generally  include 
moderate  to  low-risk  technologies  that 
should  reduce  the  intensity  and  duration  of 
the  T&E  program.  It  does,  however,  in¬ 
clude  a  requirement  for  postproduction 
test  activities  as  the  system  is  modified  to 
accommodate  previously  unknown  new 
technologies,  new  threats  or  other  perfor¬ 
mance  enhancements. 
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A  revolutionary  acquisition  strategy  incor¬ 
porates  all  the  latest  technologies  in  the 
final  production  configuration  and  is  gen¬ 
erally  a  higher-risk  approach.  As  the  con¬ 
tractor  works  on  maturing  emerging  tech¬ 
nologies,  the  T&E  work  load  increases  ir 
direct  proportion  to  the  difficulty  in  fixing 
problems.  There  is  a  much  higher  potential 
for  extended  schedules  with  iterative  test- 
fix-test  cycles. 

The  preplanned  product  improvements 
(P^I)  strategy  is  a  variant  of  the  evolution¬ 
ary  development  process  in  which  you  rec¬ 
ognize  the  high-risk  technologies /sub¬ 
systems  and  put  them  on  a  parallel  devel¬ 
opment  track.  The  testing  strategy  should 
anticipate  the  requirements  to  evaluate  P 
item  maturity  and  then  test  the  system 
during  the  integration  of  the  additional 
capability. 

Advanced  Technology  Demonstrations 
( ATD)  may  provide  early  insights  into  avail¬ 
able  technologies  for  incorporation  into 
developmental  or  mature,  post-MS  III  sys¬ 
tems.  Using  proven,  mature  technology 
provides  a  lower-risk  strategy  and  may 
significantly  reduce  the  development  test¬ 
ing  work  load  (DODI  5000.2,  5-C,  5-D). 
"Test  and  Evaluation  shall  be  used  to  deter¬ 
mine  system  maturity  and  identify  areas  of 
technical  risk,"  (part  1-C  DODD  5000.1). 
The  process  for  verifying  contract  technical 
specifications,  MIL-SPEC  and  MIL-STD 
testing  and  evaluation  of  minimum  perfor¬ 
mance  requirements  in  the  ORD,  exit  crite¬ 
ria  or  the  acquisition  program  baseline  per¬ 
formance  should  be  addressed  in  the  DT &E 
strategy.  The  DT&E  is  an  iterative  process 
starting  at  configuration  item/software 
module  levels  and  continuing  throughout 
the  component  integration  into  subassem¬ 
blies  and,  finally,  system-level  performance 
evaluations.  C^erational  test  and  evalua¬ 
tion  is  interwoven  into  early  DT&E  for 
maximizing  the  efficient  use  of  test  articles 


and  test  schedules.  However,  OT&E  must 
remain  a  distinct  thread  of  activity  that 
does  not  lose  its  identity  in  the  tapestry  of 
test  events.  Planning  for  test  resources  is 
driven  by  the  sequence  and  intensity  of 
development  test  (DT)  and  operational  test 
(OT)  events.  Resource  coordination  is  an 
equally  arduous  task,  which  frequently  has 
lead  times  equal  to  major  program  devel¬ 
opment  activities.  Included  in  the  program 
T&E  strategy  should  be  the  overshadow¬ 
ing  evaluation  plan,  outlining  methodolo¬ 
gies,  models,  simulations  and  test  data  re¬ 
quired  at  periodic  decision  points.  Part  6, 
Engineering  and  Manufacturing,  DODI 
5000.2  provides  TEMP  requirements: 

The  TEMP  will:  (a)  address  critical  human 
issues  to  provide  data  to  v  lidate  the  re¬ 
sults  of  human  factors  eng  .  leering  analy¬ 
ses;  and  (b)  require  identification  of  mis¬ 
sion  critical  operational  and  maintenance 
tasks.  [DODI  5000.2, 6-H] 

A  reliability  growth  (TAFT)  program 
should  be  developed  to  satisfy  the  re¬ 
liability  levels  required  at  MS  III.  Reli¬ 
ability  tests  and  demonstrations  (MIL- 
STD-785)  will  be  based  on  actual  or 
simulated  operational  conditions. 

Maintainability  will  be  verified  with  a 
maintainability  demonstration  (MIL- 
STD-470)  before  MS  Ill.  [DODI  5000.2, 
6-C] 

As  early  as  practicable,  developers  and 
test  agencies  will  assess  survivability 
and  validate  critical  survivability  char¬ 
acteristics  at  as  high  a  system  level  as 
possible.  The  TEMP  will  identify  the 
means  by  which  the  survivability  ob¬ 
jectives  are  validated.  [DODI  5000.2, 
6-F] 

Field  engineering  test  facilities  and  test¬ 
ing  in  the  intended  operational  envi¬ 
ronments  are  required  to  (1)  verify 
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electric  or  electronic  systems  predicted 
performance,  (2)  establish  confidence 
in  electromagnetic  compatibility  de¬ 
sign  based  on  standards  and  specifica¬ 
tions,  and  (3)  validate  electromagnetic 
compatibility  analysis  methodology. 
[DODI  5000.2, 6-G] 

The  TEMP  will  address  health  hazard  and 
safety  critical  issues  to  provide  data  to  vali¬ 
date  the  results  of  system  safety  analyses. 
[DODI  5000.2, 6-1] 

The  TEMP  strategy  should  directly  sup¬ 
port  the  development  of  more-detailed 
planning  and  resource  documents  needed 
to  execute  the  actual  test  events  and  subse¬ 
quent  evaluations. 

Test  and  Evaluation  planning  shall  address 
measures  of  performance  with  appropriate 
quantitative  criteria,  test  event  or  scenario 
description,  resource  requirements  and  test 
limitations.  Test  planning,  at  a  minimum, 
must  address  all  system  components  that 
are  critical  to  the  achievement  and  demon¬ 
stration  of  contract  technical  performance 
specifications  and  minimum  acceptable 
operational  performance  requirements 
specified  in  the  Operation  Requirements 
Document,  [part  8,  DODI  5000.2] 

17.3  TEMP  FORMAT 

The  format  specified  in  DOD  5000.2-M, 
part  7,  is  required  for  all  acquisition  cat¬ 
egory  I  and  Office  of  Secretary  of  Defense 
(OSD)  designated  oversight  programs 
(Table  17-1).  It  may  be  tailored  as  needed 
for  lesser  category  acquisition  programs  at 
the  discretion  of  the  milestone  decision 
authority.  The  TEMP  is  intended  to  be  a 
summary  document  (30  pages  in  main 
body)  outlining  DT&E  and  OT&E  manage¬ 
ment  responsibilities  across  all  phases  of 
the  acquisition  process.  When  the  develop¬ 


ment  is  a  multi-Service  or  joint  acquisition 
program,  one  integrated  TEMP  is  devel¬ 
oped  with  Service  armexes,  as  required.  A 
Capstone  TEMP  may  not  be  appropriate 
for  a  single  major  weapon  platform  but 
could  be  used  to  encompass  testing  of  a 
collection  of  individual  systems,  each  with 
its  own  annex  (e.g..  Strategic  Defense  Ini¬ 
tiative  Organization  (SDIO),  Family  of  Tac¬ 
tical  Vehicles,  FAADS).  A  program  TEMP 
is  updated  at  milestones,  upon  program 
baseline  breach  and  for  other  significant 
program  changes.  Updates  may  consist  of 
page  changes  and  are  no  longer  required 
when  a  program  has  no  further  develop¬ 
ment  activities. 

The  TEMP  is  a  living  document  that  must 
address  changes  to  critical  issues  associ¬ 
ated  with  an  acquisition  program.  Major 
changes  in  program  requirements,  sched¬ 
ule  or  funding  usually  result  in  a  change  in 
the  test  program.  Thus,  the  TEMP  must  be 
reviewed  and  updated  on  program  change, 
on  baseline  breach  and  before  each  mile¬ 
stone  decision,  to  ensure  that  T&E  require¬ 
ments  are  current.  As  the  primary  docu¬ 
ment  used  in  the  OSD  review  and  decision 
process  to  assess  the  adequacy  of  planned 
testing  and  evaluation,  the  TEMP  must  be 
of  sufficient  scope  and  content  to  explain 
the  entire  T&E  program.  The  key  topics  in 
the  TEMP  are  shown  in  Table  17-1. 

Each  TEMP  submitted  to  OSD  should  be  a 
summary  document,  detailed  only  to  the 
extent  necessary  to  show  the  rationale  for 
the  type,  amount  and  schedules  of  the  test¬ 
ing  planned.  It  must  relate  the  T&E  effort 
clearly  to  technical  risks,  operational  issues 
and  concepts,  system  performance,  reli¬ 
ability,  availability,  maintainability,  logis¬ 
tic  objectives  and  requirements,  and  major 
decision  points.  It  should  summarize  the 
testing  accomplished  to  date  and  explain 
the  relationship  of  the  various  simulations, 
subsystem  tests,  integrated  system  devel- 
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Table  17-1 .  Test  and  Evaluation  Master  Plan  Outline  (Format) 


PART  I  SYSTEM  INTRODUCTION  (2  pages  suggested  -  refer  to  annexes) 

MISSION  DESCRIPTION 
SYSTEM  THREAT  ASSESSMENT 

MINIMUM  ACCEPTABLE  OPERATIONAL  PERFORMANCE  REQUIREMENTS 

SYSTEM  DESCRIPTION 

CRITICAL  TECHNICAL  PARAMETERS 

PART  II  INTEGRATED  TEST  PROGRAM  SUMMARY  (2  cages  suggested) 

INTEGRATED  TEST  PROGRAM  SCHEDULE 
MANAGEMENT 

PART  III  DEVELOPMENT  TEST  AND  EVALUATION  OUTLINE  (10  pages  suggested) 
DEVELOPMENT  TEST  AND  EVALUATION  OVERVIEW 
DEVELOPMENT  TEST  AND  EVALUATION  TO  DATE 
FUTURE  DEVELOPMENTAL  TEST  AND  EVALUATION 
LIVE-FIRE  TEST  AND  EVALUATION 

PART  IV  OPERATIONAL  TEST  AND  EVALUATION  OUTLINE  (10  pages  suggested) 
OPERATIONAL  TEST  AND  EVALUATION  OVERVIEW 
CRITICAL  OPERATIONAL  ISSUES 
OPERATIONAL  TEST  AND  EVALUATION  TO  DATE 
FUTURE  OPERATIONAL  TEST  AND  EVALUATION 

PART  V  TEST  AND  EVALUATION  RESOURCE  SUMMARY  (6  pages  suggested) 

TEST  ARTICLES 

TEST  SITES  AND  INSTRUMENTATION 
TEST  SUPPORT  EQUIPMENT 
THREAT  SYSTEMS/SIMULATORS 
TEST  TARGETS  AND  EXPENDABLES 
OPERATIONAL  FORCE  TEST  SUPPORT 
SIMULATIONS,  MODELS  AND  TEST  BEDS 
SPECIAL  REQUIREMENTS 

TEST  AND  EVALUATION  FUNDING  REQUIREMENTS 
MANPOWEFVTRAINING 

APPENDIX  A  BIBLIOGRAPHY 
APPENDIX  B  ACRONYMS 
APPENDIX  C  POINTS  OF  CONTACT 

ANNEXES  or  ATTACHMENTS  (if  appropriate) 
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opment  tests  and  initial  operational  tests 
that,  when  analyzed  in  combination,  pro¬ 
vide  confidence  in  the  system's  readiness 
to  proceed  into  the  next  acquisition  phase. 
The  TEMP  must  address  the  T&E  to  be 
accomplished  in  each  program  phase,  with 
the  next  phase  addressed  in  the  most  detail. 
The  TEMP  is  also  used  as  a  coordination 
document  to  outline  each  test  and  support 
organization's  role  in  the  T&E  program 
and  identify  major  test  facilities  and  re¬ 
sources.  The  TEMPs  supporting  the  pro¬ 
duction  and  initial  deployment  decision 
must  include  the  T&E  planned  to  verify  the 
correction  of  deficiencies  and  to  complete 
production  qualification  testing  and  fol- 
low-on  OT&E. 

The  objective  of  the  OSD  TEMP  review 
process  is  to  ensure  successful  T&E  pro¬ 
grams  that  will  support  decisions  to  com¬ 
mit  resources  at  major  milestones.  Some  of 
the  T &E  issues  considered  during  the  TEMP 
review  process  include: 

(1)  Are  DT&E  and  OT&E  initiated  early 
to  assess  performance,  identify  risks  and 
estimate  operational  potential? 

(2)  Are  critical  issues,  test  directives  and 
evaluation  criteria  related  to  mission  need 
and  operational  requirements  established 


well  before  testing  begins? 

(3)  Are  provisions  made  for  collecting 
sufficient  test  data  with  appropriate  test 
instrumentation  to  minimize  subjective 
judgment? 

(4)  Is  OT&E  conducted  by  an  organiza¬ 
tion  independent  of  the  developer  and  user? 

(5)  Do  the  test  methodology  and  instru¬ 
mentation  provide  a  mature  and  flexible 
network  of  resources  that  stress  (as  early  as 
possible)  the  weapon  system  in  a  variety  of 
realistic  environments? 

17.4  SUMMARY 

The  PMO  is  directly  responsible  for  the 
content  and  quality  of  the  test  strategy  and 
planning  document.  The  TEMP,  as  an  inte¬ 
grated  summary  management  tool,  requires 
an  extensive  commitment  of  man-hours 
and  PM  guidance.  The  interactions  of  the 
various  T&E  players  and  support  agencies 
must  be  woven  into  the  fabric  of  the  total 
system  acquisition  strategy.  Cost  and  sched¬ 
ule  implications  must  be  negotiated  to  en¬ 
sure  a  viable  test  and  evaluation  program 
that  provides  timely  and  accurate  data  to 
the  engineering  and  management  decision¬ 
makers. 
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MODULE 

Specialized  Testing 


The  nature  of  a  weapon  system  sometimes 
requires  the  use  of  a  specially  tailored  test 
and  evaluation  program.  In  some  cases, 
hazardous  testing  must  be  performed.  In 
other  cases,  testing  must  be  conducted  by 
specialized  organizations  or  at  special  times 
in  the  development  life  cycle. 


This  module  addresses  the  testing  of  spe¬ 
cial  weapons  (such  as  chemical,  laser  and 
space  systems),  embedded  computer  sys¬ 
tems,  electronic  warfare  and  command- 
and-control  systems,  logistics  infrastruc¬ 
ture  test  and  evaluation,  and  production- 
related  testing  activities. 


18 

EMBEDDED  COMPUTER 
SYSTEMS  TESTING 


18.1  INTRODUCTION 

Software  componer\ts  present  a  major  de¬ 
velopment  risk  for  military  computer  sys¬ 
tems.  They  escalate  the  cost  and  reduce  the 
reliability  of  military  systems.  Embedded 
computer  systems  are  physically  incorpo¬ 
rated  into  larger  systems,  neither  having  a 
major  function  of  data  processing.  The  out¬ 
put  of  the  systems  are  normally  informa¬ 
tion,  control  signals  or  computer  data  re¬ 
quired  by  the  host  system  to  complete  its 
mission.  Although  hardware  and  software 
contribute  in  equal  measure  to  successful 
implementation  of  embedded  computer 
system  functions,  there  have  been  relative 
imbalances  in  their  treatment  during  sys¬ 
tem  development. 

The  development  of  embedded  systems 
involves  a  series  of  activities  in  which  there 
are  frequent  opportunities  for  errors.  Er¬ 
rors  may  occur  at  the  inception  of  the  pro¬ 
cess,  when  the  requirements  of  the  system 
may  be  erroneously  specified,  or  later  in 
the  development  cycle,  when  system  speci¬ 
fications  are  implemented.  This  chapter 
addresses  the  use  of  testing  to  control  the 
development  risk  of  embedded  computer 
systems,  particularly  as  it  pertains  to  the 
software  development  process. 

18.2  MISSION  CRITICAL  COMPUTER 
RESOURCES 

The  term  Mission  Critical  Computer  Re¬ 
sources  (MCCR)  is  defined  as  automated 


data  processing  equipment,  software  or 
services;  and  the  function,  operation  or  use 
of  the  equipment  software  or  services  in¬ 
volves: 

(1)  Intelligence  activities; 

(2)  Cryptologic  activities  related  to  na¬ 
tional  security; 

(3)  Command  and  control  of  military 
forces; 

(4)  Equipment  that  is  an  integral  part  of 
a  weapons  system; 

(5)  Critical,  direct  fulfillment  of  military 
or  intelligence  missions. 

Acquisition  of  MCCR  is  described  in  part  6, 
section  D,  DODI  5000.2,  and  directs  MCCR 
development  planning  to  follow  DOD-STD- 
2167  and  DOD-STD-2168. 

18.3  PURPOSE  OF  SOFTWARE  TEST 
AND  EVALUATION 

A  major  problem  in  software  development 
is  a  lack  of  well-defined  requirements.  If 
requirements  are  not  well-defined,  errors 
can  multiply  throughout  the  development 
process.  As  illustrated  in  Figure  18-1,  er¬ 
rors  may  occur  at  the  inception  of  the  pro¬ 
cess.  These  errors  may  occur  during  re¬ 
quirements  definition,  when  objectives  may 
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be  erroneously  or  imperfectly  specified; 
during  the  later  design  and  development 
stages,  when  these  objectives  are  imple¬ 
mented;  and  during  software  maintenance 
and  operational  phases,  when  software 
changes  are  needed  to  eliminate  errors  or 
enhance  performance.  Estimates  of  in¬ 
creased  software  costs  arising  from  incom¬ 
plete  testing  help  to  illustrate  the  dimen¬ 
sion  of  software  life-cycle  costs.  Averaged 
over  the  operational  life  cycle  of  a  com¬ 
puter  system,  development  costs  encom¬ 
pass  approximately  30  percent  of  total  sys¬ 
tem  costs.  The  remaining  70  percent  of  life- 
cycle  costs  are  associated  with  maintenance, 
which  includes  system  enhancements  and 
error  correction.  Complete  testing  during 
earlier  development  phases  may  have  de¬ 
tected  these  errors.  The  relative  costs  of 
error  correction  increase  as  a  function  of 
time  from  the  start  of  the  development 
process.  Relative  costs  of  error  correction 
rise  dramatically  between  requirements  and 
design  phases  and  more  dramatically  dur¬ 
ing  code  implementation. 

Previous  research  in  the  area  of  software 
T&E  reveals  that  half  of  all  maintenance 
costs  are  incurred  in  the  correction  of  previ¬ 
ously  undetected  errors.  Approximately 
one-half  of  the  operational  life  cycle  costs 
can  be  traced  directly  to  inadequate  or  in¬ 
complete  testing  activities.  In  addition  to 
cost  increases,  operational  implications  of 
software  errors  in  weapon  systems  can  re¬ 
sult  in  mission  critical  software  failures 
that  may  impact  mission  success  and  per¬ 
sonnel  safety. 

A  more  systematic  and  rigorous  approach 
to  software  testing  is  required.  To  be  effec¬ 
tive,  this  approach  must  be  applied  to  all 
phases  of  the  development  process  in  a 
planned  and  coordinated  manner,  begin¬ 
ning  at  the  earliest  design  stages  and  pro¬ 
ceeding  through  operational  testing  of  the 
integrated  system.  Early,  detailed  software 


test  and  evaluation  (T&E)  plaiming  is  criti¬ 
cal  to  the  successful  development  of  a  com¬ 
puter  system. 

18.4  SOFTWARE  DEVELOPMENT 
PROCESS 

Software  engineering  technologies  used  to 
produce  operational  software  are  key  risk 
factors  in  a  development  program.  The  T&E 
program  should  help  determine  which  of 
these  technologies  increase  risk  and  have  a 
life-cycle  impact.  A  principal  source  of  risk 
is  the  support  software  required  to  develop 
operational  software.  In  terms  of  life-cycle 
impact,  operational  software  problems  are 
commonly  associated  with  the  difficulty  in 
maintaining  and  supporting  the  software 
once  deployed.  Software  assessment  re¬ 
quires  an  analysis  of  the  life-cycle  impact, 
which  varies  depending  on  the  technology 
used  to  design  and  implement  the  soft¬ 
ware.  One  approach  to  reducing  long-term 
life-cycle  risks  is  to  use  ADA  language  and 
common  hardware  throughout  the  devel¬ 
opment  and  operation  of  the  software. 
Ihese  life-cycle  characteristics  that  affect 
operational  capabilities  must  be  addressed 
in  the  Test  and  Evaluation  Master  Plan 
(TEMP),  and  tests  should  be  developed  to 
identify  problems  caused  by  these  char¬ 
acteristics.  The  technology  used  to  design 
and  implement  the  software  may  signifi¬ 
cantly  affect  software  supportability  and 
maintainability. 

The  TEMP  must  sufficiently  describe  the 
acceptance  criteria  or  software  metrics  from 
the  written  specifications  for  operational 
effectiveness  and  suitability.  The  specifica¬ 
tions  must  define  the  required  software 
metrics  to  set  objectives  and  thresholds  for 
mission  critical  functions.  Additionally, 
these  metrics  should  be  evaluated  at  the 
appropriate  stage  of  system  development 
rather  than  at  some  arbitrarily  imposed 
milestone. 
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18.5  T&E  IN  THE  SOFTWARE 
LIFE  CYCLE 

Software  testing  is  an  iterative  process  ex¬ 
ecuted  at  all  development  stages  to  exam¬ 
ine  program  design  and  code  to  expose 
errors.  Software  test  planning  should  be 
described  in  the  TEMP  with  the  same  care 
as  test  planning  for  other  system  compo¬ 
nents. 

18.5.1  Testing  Approach 

The  integration  of  software /MCCR  devel¬ 
opment  into  the  overall  acquisition  process 
dictates  a  testing  process  consistent  with 
the  bottom-up  approach  taken  with  hard¬ 
ware  development.  The  earliest  stage  of 
software  testing  is  characterized  by  heavy 
human  involvement  in  basic  design  and 
coding  processes.  Thus,  human  testing  is 
defined  as  informal,  noncomputer-based 
methods  of  evaluating  architectures,  de¬ 
signs  and  interfaces.  It  can  consist  of: 

•  Inspections  —  The  programmer  explains 
his  work  to  a  small  group  of  peers  with 
discussion  and  direct  feedback  on  errors, 
inconsistencies  and  omissions. 

•  Walk-through  —  A  group  of  peers 
develop  test  cases  to  evaluate  work  to  date 
and  give  direct  feedback  to  the  program¬ 
mer. 

•  Desk  Checking  —  A  self  evaluation  is 
made  by  the  programmer  of  his  work.  There 
is  a  low  probability  of  identifying  his  errors 
of  logic  or  coding. 

•  Peer  Ratings  —  Mutually  supportive, 
anonymous  reviews  are  performed  by 
groups  of  peers  with  collaborative  evalua¬ 
tions  and  feedback. 

•  Design  Reviews  —  Preliminary  design 
reviews  (PDRs)  and  critical  design  reviews 


(CDRs)  provide  milestones  in  the  develop¬ 
ment  efforts  that  review  development  and 
evaluations  to  date.  An  independent  verifi¬ 
cation  and  validation  (IV&V)  contractor 
may  facilitate  the  government's  ability  to 
give  meaningful  feedback. 

Once  the  development  effort  has  matured 
beyond  the  benefits  of  human  testing,  com¬ 
puterized-software-only  testing  may  be 
appropriate.  It  is  performed  to  determine 
the  functionality  of  the  software  when  tested 
as  an  entity  or  "build . "  Documentation  con¬ 
trol  is  essential  so  that  test  results  are  corre¬ 
lated  with  the  appropriate  version  of  the 
build.  Software  testing  is  usually  conducted 
using  some  combination  of  "black  box" 
and  "white  box"  testing. 

•  Black  Box  —  Functional  testing  of  a 
software  unit  without  knowledge  of  how 
the  internal  structure  or  logic  will  process 
the  input  to  obtain  the  specified  output. 
Within-boundary  and  out-of-boundary 
stimulants  test  the  software's  ability  to 
handle  abnormal  events.  Most  likely  cases 
are  tested  to  provide  a  reasonable  assur¬ 
ance  that  the  software  will  demonstrate 
specified  performance.  Even  the  simplest 
software  designs  rapidly  exceed  our  capac¬ 
ity  to  test  all  alternatives. 

•  White  Box  —  Structural  testing  of  the 
internal  logic  and  software  structure  pro¬ 
vides  an  opportunity  for  more  extensive 
identification  and  testing  of  critical  paths. 
The  process  and  objectives  are  otherwise 
very  similar  to  black  box  testing. 

Testing  should  be  performed  from  the  bot¬ 
tom  up.  The  smallest  controlled  software 
modules  —  computer  software  units  —  are 
tested  individually.  They  are  then  com¬ 
bined  or  integrated  and  tested  in  larger 
aggregate  groups  or  builds.  When  this  pro¬ 
cess  is  complete,  the  software  system  is 
tested  in  its  entirety.  Obviously,  as  errors 
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are  found  in  the  latter  stages  of  the  test 
program,  a  return  to  earlier  portions  of  the 
development  program  to  provide  correc¬ 
tions  is  required.  The  cost  impact  of  error 
detection  and  correction  can  be  diminished 
using  the  bottom-up  testing  approach. 

System  level  testing  can  begin  once  all 
modules  in  the  computer  software  con¬ 
figuration  item  (CSCI)  have  been  coded 
and  individually  tested.  A  software  inte¬ 
gration  lab  (SIL),  with  adequate  machine 
time  and  appropriate  simulations,  will  fa¬ 
cilitate  hardware  simulation /emulation 
and  the  operating  environment.  If  data 
analysis  indicates  proper  software  func¬ 
tioning,  it  is  time  to  advance  to  a  more 
complex  and  realistic  test  environment. 

•  Hot  Bench  Testing  —  Integration  of 
the  software  released  from  the  SIL  for  full- 
up  testing  with  actual  system  hardware  in 
a  hardware-in-the-loop  (HWIL)  facility 
marks  a  significant  advance  in  the  develop¬ 
ment  process.  Close  approximation  of  the 
actual  operating  environment  should  pro¬ 
vide  test  sequences  and  stress  needed  to 
evaluate  the  effectiveness  of  the  software 
system(s).  Problems  stimulated  by  the 
"noisy  environment,"  interface  problems, 
electromagnetic  interference  (EMI)  and  dif¬ 
ferent  electrical  transients  should  surface. 
Good  hardware  and  software  test  programs 
leading  up  to  HWIL  testing  should  aid  in 
isolating  problems  to  the  hardware  or  soft¬ 
ware  side  of  the  system.  Caution  should  be 
taken  to  avoid  any  outside  stimuli  that 
might  trigger  unrealistic  responses. 

•  Field  T esting — Development  test  and 
evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E)  events  must  be  designed 
to  provide  for  data  collection  processes  and 
instrumentation  that  will  measure  system 
responses  and  allow  data  analysts  to  iden¬ 
tify  the  appropriate  causes  of  malfunctions. 
Field  testing  should  be  rigorous,  providing 
environmental  stresses  and  mission  pro¬ 


files  likely  to  be  encountered  in  operational 
scenarios.  Government  software  support 
facilities  tasked  for  future  maintenance  of 
the  software  system  should  be  brought  on 
board  to  familiarize  them  with  the  system 
operating  characteristics  and  documenta¬ 
tion.  Their  expertise  should  be  included  in 
the  software  test  and  evaluation  process  to 
assist  in  the  selection  of  stimuli  likely  to 
expose  sf  ftware  problems. 

It  is  critical  that  adequate  software  T&E 
information  be  contained  in  documents 
such  as  TEMPs  and  test  plans.  The  TEMP 
must  define  characteristics  of  critical  soft¬ 
ware  components  that  effectively  address 
goals  and  thresholds  for  mission  critical 
functions.  The  measures  of  effectiveness 
(MOEs)  must  support  the  critical  software 
issues.  The  test  plan  should  specify  the  test 
methodologies  that  will  be  applied.  Test 
methodologies  consist  of  two  components. 
The  first  is  the  test  strategy  that  guides  the 
overall  testing  effort,  and  the  second  is  the 
testing  technique  that  is  applied  within  the 
framework  of  a  test  strategy. 

Effective  test  methodologies  require  realis¬ 
tic  software  test  environments  and  sce¬ 
narios.  The  test  scenarios  must  be  appro¬ 
priate  for  the  test  objectives;  i.e.,  the  test 
results  must  be  interpretable  in  terms  of 
software  test  objectives.  The  test  scenarios 
and  analysis  should  actually  verify  and 
validate  accomplishment  of  requirements. 
The  test  environments  must  be  chosen  on  a 
careful  analysis  of  characteristics  to  be  dem- 
onstrated  and  its  relationship  to  the  devel¬ 
opment,  operational  and  support  environ¬ 
ments.  In  addition,  environment  must  be 
representative  of  that  in  which  the  soft¬ 
ware  will  be  maintained. 

18.5  2  Independent  Verification 
and  Validation 

Independent  verification  and  validation  are 
risk-reducing  techniques  that  are  applied 
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to  major  software  development  efforts.  The 
primary  purpose  of  IV &V  is  to  ensure  that 
software  meets  requirements  and  is  reli¬ 
able  and  maintainable.  The  IV&V  is  effec¬ 
tive  only  if  implemented  early  in  the  soft¬ 
ware  development  schedule.  Requirements 
analysis  and  risk  assessment  are  the  most 
critical  activities  performed  by  IV&V  orga¬ 
nizations;  their  effectiveness  is  limited  if 
brought  on  board  a  project  after  the  fact. 
Often,  there  is  a  reluctance  to  implement 
IV&V  because  of  the  costs  involved,  but 
early  implementation  of  IV&V  will  result 
in  lower  overall  costs  of  error  correction 
and  software  maintenance.  As  develop¬ 
ment  efforts  progress,  IV&V  involvement 
typically  decreases.  This  is  due  more  to  the 
expense  of  continued  involvement  than  to 
a  lack  of  need.  For  an  IV&V  program  to  be 
effective,  it  must  be  the  responsibility  of  an 
individual  or  organization  external  to  the 
software  development  program  manager. 

The  application  of  the  IV&V  process  to 
software  development  maximizes  the  main¬ 
tainability  of  the  fielded  software  system, 
while  minimizing  the  cost  of  developing 
and  fielding  it.  Maintenance  of  a  software 
system  falls  into  several  major  categories: 
corrective  maintenance,  modifying  soft¬ 
ware  to  correct  errors  in  operation;  adap¬ 
tive  maintenance,  modifying  the  software 
to  meet  changing  requirements;  and  per¬ 
fective  maintenance,  modifying  the  soft¬ 
ware  to  incorporate  new  features  or  im¬ 
provements. 

The  rV&V  process  maximizes  the  reliabil¬ 
ity  of  the  software  product,  which  eases  the 
performance  of  and  minimizes  the  need  for 
corrective  maintenance.  It  attempts  to  maxi- 
mize  the  flexibility  of  the  software  product, 
which  eases  the  performance  of  adaptive 
and  perfective  maintenance.  These  goals 
are  achieved  primarily  by  determining  at 
each  step  of  the  software  development  pro¬ 
cess  that  the  software  product  completely 


and  correctly  meets  the  specific  require¬ 
ments  determined  at  the  previous  step  of 
development.  This  step-by-step,  iterative 
process  continues  from  the  initial  defini¬ 
tion  of  system  performance  requirements 
through  final  acceptance  testing. 

The  review  of  software  documentation  at 
each  stage  of  development  is  a  major  por¬ 
tion  of  the  verification  process.  The  current 
documentation  is  a  description  of  the  soft¬ 
ware  product  at  the  present  stage  of  devel¬ 
opment  and  will  define  the  requirements 
laid  on  the  software  product  at  the  follow¬ 
ing  stage.  Careful  examination  and  analy¬ 
sis  of  the  development  documentation  en¬ 
sure  that  each  step  in  the  software  design 
process  is  consistent  with  the  previous  step. 
Omissions,  inconsistencies  or  design  er¬ 
rors  can  then  be  identified  and  corrected 
early  in  the  devel  pment  process. 

Continuing  participation  in  formal  and  in¬ 
formal  design  reviews  by  the  FV&V  organi¬ 
zation  maintains  the  communication  flow 
between  software  system  "customers"  and 
developers,  ensuring  that  software  design 
and  production  proceed  with  minimal  de¬ 
lays  and  misunderstandings.  Frequent  in¬ 
formal  reviews,  design  and  code  walk¬ 
through  and  audits  ensure  that  the  pro¬ 
gramming  standards,  software  engineer¬ 
ing  standards,  software  quality  assurance 
and  configuration  management  procedures 
designed  to  produce  a  reliable,  maintain¬ 
able  operational  software  system  are  fol¬ 
lowed  throughout  the  process.  Continuous 
monitoring  of  computer  hardware  resource 
allocation  throughout  the  software  devel¬ 
opment  process  also  ensures  that  the  fielded 
system  has  adequate  capacity  to  meet  op¬ 
eration  and  maintainability  requirements. 

The  entire  testing  process,  from  the  plan¬ 
ning  stage  through  final  acceptance  test,  is 
also  approached  in  a  step-by-step  manner 
by  the  FV&V  process.  At  each  stage  of  de- 
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velopment,  the  functional  requirements 
determine  test  criteria  as  well  as  design 
criteria  for  the  next  stage.  An  important 
function  of  the  IV&V  process  is  to  ensure 
that  the  test  requirements  are  derived  di¬ 
rectly  from  the  performance  requirements 
and  are  independent  of  design  implemen¬ 
tation.  Monitoring  of,  participation  in  emd 
performance  of  the  various  testing  and  in¬ 
spection  activities  by  the  IV&V  contractor 
ensure  that  the  developed  software  meets 
requirements  at  each  stage  of  development. 

Throughout  the  software  development  pro¬ 
cess,  the  rV&V  contractor  reviews  any  pro¬ 
posals  for  software  enhancement  or  change, 
proposed  changes  in  development 
baselines,  and  proposed  solutions  to  de¬ 
sign  or  implementation  problems  to  ensure 
that  the  original  performance  requirements 
are  not  forgotten.  An  important  facet  of  the 
IV&V  contractor's  role  is  to  act  as  the  objec¬ 
tive  third  party,  continuously  maintaining 
the  "audit  trail"  from  the  initial  perfor¬ 


mance  requirements  to  the  final  operational 
system. 

18.6  SUMMARY 

There  is  a  useful  body  of  software  testing 
technologies  that  can  be  applied  to  testing 
of  embedded  systems.  As  a  technical  disci¬ 
pline,  though,  software  testing  is  still  ma¬ 
turing.  There  is  little  to  guide  the  program 
manager  in  choosing  one  testing  technique 
over  another.  It  is  apparent  that  systematic 
T&E  techniques  are  far  superior  to  ad-hoc 
testing  techniques.  Implementation  of  an 
effective  T&E  plan  requires  a  set  of  strong 
technical  and  management  controls.  Given 
the  increasing  number  of  embedded  com¬ 
puter  systems  being  acquired,  there  will  be 
an  increased  emphasis  on  tools  and  tech¬ 
niques  for  T&E.  For  more-detailed  infor¬ 
mation  on  MCCR  development  and  test¬ 
ing,  review  the  DSMC  Mission  Critical 
Computer  Resource  Management  Guide. 
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Figure  18-2.  System  Development  Process 
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TESTING  FOR  VULNERABILITY 
AND  LETHALITY 


19.1  INTRODUCTION 

This  chapter  addresses  the  need  to  explore 
the  vulnerability  and  lethality  aspects  of 
a  system  through  test  and  evaluation  (T&E) 
practices  and  procedures.  In  particular, 
this  chapter  describes  the  legislatively- 
mandated  Live  Fire  Test  Program,  which 
has  been  established  to  evaluate  the  vul¬ 
nerability  and  lethality  of  developing  sys¬ 
tems.  It  also  discusses  the  role  of  T&E  in 
assessing  a  system's  ability  to  perform  in  a 
nuclear  combat  environment.  The  discus¬ 
sion  of  testing  for  nuclear  survivability  is 
based  primarily  on  information  contained 
in  the  "Nuclear  Survivability  Handbook 
for  OT&E,"  prepared  by  the  Air  Force  Op¬ 
erational  Test  and  Evaluation  Center  (Refer¬ 
ence  91). 

19.2  LIVE  FIRE  TESTING 
19.2.1  Background 

In  March  1984,  OSD  chartered  a  joint  T&E 
program  designated  "The  Joint  Live  Fire 
Program."  This  program  was  to  assess  the 
vulnerabilities  and  lethalities  of  selected 
U.S.  and  threat  systems  already  fielded. 
The  controversy  over  joint  live  fire  testing 
of  the  Army's  Bradley  Fighting  Vehicle 
System,  subsequent  congressional  hearings 
and  media  exposure  resulted  in  provisions 
being  incorporated  in  the  National  Defense 
Authorization  Act  of  FY  1987.  This  act 
required  an  Office  of  the  Secretary  of  De¬ 
fense  (OSD)-managed  Live  Fire  Testing 


(LFT)  program  for  major  acquisition  pro¬ 
grams  fitting  certain  criteria.  Subsequent 
amendments  to  legislative  guidance  have 
dictated  the  current  program.  The  DOD 
implementation  of  congressional  guidance 
in  DODI  5000.2,  part  8,  requires  that  "cov¬ 
ered  major  systems  and  munitions  pro¬ 
grams,"  (i  e..  Acquisition  Category  (AC  AT) 
I  and  II  programs)  must  execute  survivabil¬ 
ity  and  lethality  testing  before  Milestone 
(MS)  III,  full-rate  production.  Addition¬ 
ally,  product  improvements  to  those  sys¬ 
tems  may  reinitiate  live  fire  testing  require¬ 
ments.  The  Secretary  of  Defense  has  the 
authority  to  waive  these  requirements  be¬ 
fore  the  system  passes  MS  II.  Programs 
subject  to  LFT  are  listed  on  the  OSD  annual 
oversight  list.  The  USD(A)  agent  for  man¬ 
agement  of  the  DOD  Live  Fire  Test  pro¬ 
gram  is  the  Director,  Development  Test. 
This  type  of  development  test  and  evalua¬ 
tion  (DT&E)  must  be  planned  to  start  early 
enough  in  the  development  process  to  im¬ 
pact  design  and  to  provide  timely  test  data 
for  the  OSD  Live  Fire  Test  Report  required 
for  the  MS  III  Defense  Acquisition  Board 
(DAB)  and  congressional  committees.  The 
Service-detailed  Live  Fire  Test  Plan  must 
be  reviewed  and  approved  by  the  Director, 
Test  and  Evaluation  (DTE),  and  live  fire 
testing  must  be  addressed  in  Part  III  of  the 
program  Test  and  Evaluation  Master  Plan 
(TEMP).  The  OSD  has  published  guide¬ 
lines,  which  can  be  obtained  from  the  DTE. 
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19.2.2  Live  Fire  Tests 

There  are  varying  types  and  degrees  of  live 
fire  tests.  The  matrix  in  Table  19-1  illus¬ 
trates  the  various  possible  combinations. 
Full-scale,  full-up  testing  is  usually  consid¬ 
ered  to  be  the  most  realistic  and  is  the  type 
of  testing  called  for  in  the  National  Defense 
Authorization  Act  for  FY  1987. 

The  importance  of  full-scale  testing  has 
been  well  demonstrated  by  the  Joint  Live 
Fire  QLF)  tests.  In  one  case,  these  tests 
contradicted  earlier  conclusions  concern¬ 
ing  the  flammability  of  a  new  hydraulic 
fluid  used  in  F-15  and  F-16  aircraft.  Labora¬ 
tory  tests  had  demonstrated  that  the  new 
fluid  was  less  flammable  than  the  standard 
fluid.  However,  during  the  JLF  tests,  30 
percent  of  the  shots  on  the  new  fluid  re¬ 
sulted  in  fires  contrasted  with  15  percent  of 
the  shots  on  the  standard  fluid  (Reference 
100). 

While  much  insight  and  valuable  wisdom 
are  to  be  obtained  through  the  testing  of 
components  or  subsystems,  some  phenom¬ 
ena  are  only  observable  when  full-up  sys¬ 
tems  are  tested.  The  interaction  of  such 
phenomena  has  been  termed  "cascading 
damage."  Such  damage  is  a  result  of  the 
synergistic  damage  mechanisms  that  are  at 
work  in  the  "real  world"  and  likely  to  be 
found  during  actual  combat.  Live  Fire  Test¬ 
ing  provides  a  way  of  examining  the  dam¬ 
ages  inflicted  not  only  on  materiel  but  also 
on  personnel.  The  crew  casualty  problem  is 
an  important  issue  that  the  LFT  program  is 
addressing.  The  program  provides  an  op¬ 
portunity  to  assess  the  effects  of  the  com¬ 
plex  environments  that  crews  are  likely  to 
encounter  in  combat  (e.g.,  fire,  toxic  fumes, 
blunt  injury  shock  and  acoustic  injuries) 
(Reference  91). 

19.2.3  Use  of  Modeling  and  Simulation 

Survivability  and  lethality  assessments 
have  traditionally  relied  largely  on  the  use 


of  modeling  and  simulation  techniques. 
The  Live  Fire  Test  Program  does  not  re¬ 
place  the  need  for  such  techniques;  in  fact, 
the  Live  Fire  Test  Guidelines  issued  by 
OSD  in  May  1987  require  that  no  shots  be 
conducted  until  pre-shot  model  predictions 
are  made  concerning  the  expected  damage. 
Such  predictions  are  useful  for  several  rea¬ 
sons.  First,  they  assist  in  the  test  planning 
process.  If  a  model  predicts  that  no  damage 
will  be  inflicted,  test  designers  and  plan¬ 
ners  should  reexamine  the  selection  of  the 
shotlines  and/or  reassess  the  accuracy  of 
the  threat  representation.  Second,  pre-shot 
model  predictions  provide  the  Services  with 
the  opportunity  to  validate  the  accuracy  of 
the  models  by  comparing  them  with  actual 
LFT  results.  At  the  same  time,  the  LFT 
program  reveals  areas  of  damage  that  may 
be  absent  from  existing  models  and  simu¬ 
lations.  Third,  pre-shot  model  predictions 
can  be  used  to  help  conserve  scarce  target 
resources.  For  example,  models  can  be  used 
to  determine  a  sequence  of  shots  that  pro¬ 
vides  for  the  less-damaging  shots  to  be 
conducted  first,  followed  by  the  more-cata¬ 
strophic  shots  resulting  in  maximum  target 
damage. 

19.2.4  Live  Fire  Test  Guidelines 

The  Live  Fire  Test  Planning  Guide  was 
updated  by  OSD  in  June  1989.  The  guide¬ 
lines  state  that  plans  for  live  fire  testing 
must  be  included  in  the  TEMP.  Key  points 
covered  in  the  LFT  guidelines  include  the 
following: 

•  The  Live  Fire  Test  and  Evaluation 
(LFT&E)  plan  is  the  basic  planning  docu¬ 
ment  used  by  OSD  and  the  ^rvices  to  plan, 
review  and  approve  LFT&E. 

•  The  LFT&E  plan  must  contain  general 
information  on  the  system's  required  per¬ 
formance,  operational  and  technical  char¬ 
acteristics,  critical  test  objectives  and  the 
evaluation  process. 
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Table  19-1 .  Types  of  Live  Rre  Testing 
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•  Each  LFT&E  plan  must  include  testing 
of  complete  systems.  A  limited  set  of  live 
fire  tests  may  involve  production  compo¬ 
nents  configured  as  a  subsystem  before 
full-up  testing. 

•  A  Service  report  must  be  submitted 
within  60  days  of  the  completion  of  the  live 
fire  test.  The  report  must  include  the  firing 
results,  test  conditions,  limitations  and  con¬ 
clusions  and  be  submitted  in  classified  and 
unclassified  form. 

•  Within  45  days  of  receipt  of  the  Service 
report,  a  separate  Live  Fire  Test  Report 
(part  10,  EXDD  5000.2-M)  will  be  produced 
by  OSD.  The  conclusions  of  the  OSD  report 
will  be  independent  of  the  conclusions  of 
the  Service  report. 

•  The  Congress  shall  have  access  to  all 
live  fire  test  data  and  all  live  fire  test  reports 
held  by  or  produced  by  the  Secretary  of  the 
concerned  Service  or  by  OSD. 

•  The  costs  of  all  live  fire  tests  shall  be 
paid  from  funding  for  the  system  being 
tested.  In  some  instances,  the  DTE  may 
elect  to  supplement  such  funds  for  the  ac¬ 
quisition  of  targets  or  target  simulators, 
although  the  ultimate  responsibility  rests 
on  the  concerned  Service. 

19.3  TESTING  FOR  NUCLEAR 
HARDNESS  AND  SURVIVABILITY 

19.3.1  Background 

Nuclear  survivability  must  be  incorporated 
into  the  design,  acquisition  and  operation 
of  all  systems  that  must  perform  critical 
missions  in  a  nuclear  environment.  Nuclear 
survivability  is  achieved  through  a  combi¬ 
nation  of  four  methods:  hardness,  avoid¬ 
ance,  proliferation  and  reconstitution. 
Hardness  allows  a  system  to  physically 
withstand  a  nuclear  attack.  Avoidance 


encompasses  measures  taken  to  avoid  en¬ 
countering  a  nuclear  environment.  Prolif¬ 
eration  involves  having  sufficient  systems 
to  compensate  for  probable  losses.  Recon¬ 
stitution  includes  the  actions  taken  to  re¬ 
pair  or  resupply  damaged  units  in  time  to 
complete  a  mission  satisfactorily. 

There  is  a  wide  variety  of  possible  effects 
from  a  nuclear  detonation.  They  include: 
electromagnetic  pulse  (EMP),  ionizing  ra¬ 
diation,  thermal  radiation,  blast,  shock, 
dust,  debris,  blackout  and  scintillation.  Each 
weapon  system  is  susceptible  to  some  but 
not  all  of  these  effects.  The  program  man¬ 
ager  and  his  staff  must  identify  the  effects 
that  may  have  an  impact  on  the  system 
under  development  and  manage  the  de¬ 
sign,  development  and  testing  of  the  sys¬ 
tem  in  a  manner  that  minimizes  degrada¬ 
tion.  The  variety  of  possible  nuclear  effects 
is  described  more  fully  in  the  "Nuclear 
Survivability  Handbook  for  Air  Force 
OT&E"  (Reference  91). 

19.3.2  Assessing  Nuclear  Survivability 
Throughout  The  System  Acquisition 
Cycle 

The  program  manager  must  ensure  that 
nuclear  survivability  issues  are  addressed 
throughout  the  system  acquisition  cycle. 
During  the  Concept  Exploration  and  Defi¬ 
nition  Phase,  the  survivability  requirements 
stated  in  the  Service  requirements  docu¬ 
ment  should  be  verified,  refined  or  further 
defined.  "Critical  survivability  character¬ 
istics  will  be  used  to  evolve  survivability 
design  criteria  which  will  be  included  in 
appropriate  configuration  baselines"  (part 
6-F,  DODI  5000.2).  During  the  Demonstra¬ 
tion  and  Validation  Phase,  trade-offs  be¬ 
tween  hardness  levels  and  other  system 
characteristics  (such  as  weight, 
decontaminability  and  compatibility) 
should  be  described  quantitatively.  Trade¬ 
offs,  between  hardness,  avoidance,  prolif- 


eration  and  reconstitution  as  a  method  for 
achieving  survivability,  should  also  be  con¬ 
sidered  at  this  time.  During  the  Engineer¬ 
ing  and  Manufacturing  Development 
Phase,  the  system  must  be  adequately  tested 
to  confirm  that  hardness  objectives,  crite¬ 
ria,  requirements  and  specifications  are  met. 
Plans  for  nuclear  hardness  and  survivabil¬ 
ity  testing  must  be  outlined  in  the  TEMP 
(part  6-F,  DODI  5000.2).  The  appropriate 
commands  must  make  provision  for  test 
and  hardness  surveillance  equipment  and 
procedures  so  required  hardness  levels  can 
be  maintained  once  the  system  is  opera¬ 
tional. 

During  the  Production  and  Deployment 
Phase,  system  hardness  is  maintained 
through  an  active  hardness  assurance  pro¬ 
gram.  Such  a  program  ensures  that  the  end 
product  conforms  to  hardness  design  speci¬ 
fications  and  that  hardness  aspects  are  re¬ 
evaluated  before  any  retrofit  changes  are 
made  to  existing  systems. 

Once  a  system  is  operational,  a  hardness 
surveillance  program  may  be  implemented 
to  maintain  system  hardness  and  to  iden¬ 
tify  any  further  evaluation,  testing  or  retro¬ 
fit  changes  required  to  ensure  survivabil¬ 
ity.  A  hardness  surveillance  program  con¬ 
sists  of  a  set  of  scheduled  tests  and  inspec¬ 
tions  to  ensure  that  a  system's  designed 
hardness  is  not  degraded  through  opera¬ 
tional  use,  logistic  support,  maintenance 
actions  or  natural  causes. 

19.3.3  Test  Planning 

The  "Nuclear  Survivability  Handbook  for 
Air  Force  OT&E"  describes  the  following 
challenges  associated  with  nuclear  hard¬ 
ness  and  survivability  testing: 

(1)  The  magnitude  and  range  of  effects 
from  a  nuclear  burst  are  much  greater  than 
those  from  conventional  explosions  that 


may  be  used  to  simulate  nuclear  bursts. 
Nuclear  detonations  have  effects  not  found 
in  conventional  explosions.  The  intense 
nuclear  radiation,  blast,  shock,  thermal  and 
EMP  fields  are  difficult  to  simulate.  In  ad¬ 
dition,  systems  are  often  tested  at  stress 
levels  that  are  either  lower  than  those  es¬ 
tablished  by  the  criteria  or  lower  than  the 
level  needed  to  cause  damage  to  the  sys¬ 
tem. 

(2)  The  yields  and  configurations  for  un¬ 
derground  testing  are  limited.  It  is  gener¬ 
ally  not  possible  to  test  all  relevant  effects 
simultaneously  or  to  observe  possibly  im¬ 
portant  synergism  between  effects. 

(3)  System-level  testing  for  nuclear  ef¬ 
fects  is  normally  expensive,  takes  years  to 
plan  and  conduct  and  requires  specialized 
expertise.  Often,  classes  of  tests  conducted 
early  in  the  program  are  not  repeated  later. 
Therefore,  operational  requirements  should 
be  folded  into  these  tests  from  the  start, 
often  early  in  the  acquisition  process.  This 
mandates  a  more-extensive,  combined 
DT&E/OT&E  test  program  than  normally 
found  in  other  types  of  testing. 

Program  managers  and  test  managers  must 
remain  sensitive  to  the  ambiguities  involved 
in  testing  for  nuclear  survivability.  For  ex¬ 
ample,  there  is  no  universal  quantitative 
measure  of  survivability;  and  statements  of 
survivability  may  lend  themselves  to  a  va¬ 
riety  of  interpretations.  Moreover,  it  can  be 
difficult  to  combine  system  vulnerability 
estimates  for  various  nuclear  effects  into  an 
assessment  of  overall  survivability.  As  a 
result,  program/ test  managers  must  exer¬ 
cise  caution  when  developing  test  objec¬ 
tives  and  specifying  measures  of  merit  re¬ 
lated  to  nuclear  survivability. 

19.3.4  Test  Execution 

For  nuclear  hardness  and  survivability  test¬ 
ing,  development  test  (DT)  and  operational 
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test  (OT)  efforts  are  often  combined  be¬ 
cause  it  is  not  possible  to  test  in  an  opera¬ 
tional  nuclear  environment.  The  use  of  an 
integrated  DT/OT  program  requires  early 
and  continuous  dialogue  between  the  two 
test  communities  so  each  understands  the 
needs  of  the  other  and  maximum  coopera¬ 
tion  in  meeting  objectives  is  obtained. 

Test  and  evaluation  techniques  available  to 
validate  the  nuclear  survivability  aspects 
of  systems  and  subsystems  include  under¬ 
ground  nuclear  testing,  environmental 
simulation  (system  level,  subsystem  level 
and  component  level)  and  analytical  simu¬ 
lation.  Table  19-3  outlines  the  major  activi¬ 
ties  relevant  to  the  assessment  of  nuclear 
hardness  and  survivability  and  the  phases 
of  the  acquisition  cycle  in  which  they  occur. 

19.4  SUMMARY 

The  vulnerability  and  lethality  aspects  of  a 
system  can  be  evaluated  through  live  fire 


tests.  These  tests  are  used  to  provide  an 
insight  into  the  system's  ability  to  protect 
its  crew  and  to  continue  to  operate/ fight 
after  being  hit  by  enemy  weapons.  It  pro¬ 
vides  a  way  of  examining  the  damages 
inflicted  not  only  on  materiel  but  also  on 
personnel.  Live  fire  testing  also  provides 
an  opportunity  to  assess  the  effects  of  com¬ 
plex  environments  that  crews  are  likely  to 
encounter  in  combat. 

Nuclear  survivability  must  be  carefully 
evaluated  during  the  system  acquisition 
cycle.  Trade-offs  between  hardness  levels 
and  other  system  characteristics,  such  as 
weight,  speed,  range,  cost,  etc.,  must  be 
evaluated.  Nuclear  survivability  testing  is 
difficult,  and  the  evaluation  of  test  results 
may  lend  itself  to  a  variety  of  interpreta¬ 
tions.  Therefore,  program  managers  must 
exercise  caution  when  developing  test  ob¬ 
jectives  related  to  nuclear  survivability. 
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Table  19-2.  Relationships  Between  Key  Concepts 
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Table  1 9-3.  Nuclear  Hardness  and  Survivability  Assessment  Activities  (Continued) 
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20 

LOGISTICS  INFRASTRUCTURE 

T&E 


20.1  INTRODUCTION 

In  all  materiel  acquisition  programs,  the 
integrated  logistics  support  (ILS)  effort  be¬ 
gins  in  the  Mission  Area  Analysis  Phase 
before  program  initiation,  continues 
throughout  the  acquisition  cycle  and  ex¬ 
tends  past  the  deployment  phase.  Logistics 
testing  must,  therefore,  extend  over  the 
entire  acquisition  cycle  of  the  system  and 
be  carefully  planned  and  executed  to  en¬ 
sure  the  readiness  tmd  supportability  of  the 
system.  This  chapter  covers  the  develop¬ 
ment  of  logistics  support  test  requirements 
and  the  conduct  of  supportability  assess¬ 
ments  to  ensure  that  readiness  and  sup¬ 
portability  objectives  are  identified  and 
achieved.  The  importance  of  the  ILS 
manager's  participation  in  the  Test  and 
Evaluation  Master  Plan  (TEMP)  develop¬ 
ment  process  should  be  stressed.  He  must 
ensure  the  ILS  test  and  evaluation  (T&E) 
objectives  are  considered  and  that  adequate 
resources  are  available  for  ILS  T&E. 

Integrated  logistic  support  is  defined  as  a 
disciplined,  unified  and  iterative  approach 
to  the  management  and  technical  activities 
necessary  to  integrate  support  consider¬ 
ations  into  system  and  equipment  design; 
develop  support  requirements  that  are  re¬ 
lated  consistently  to  readiness  objectives, 
design  and  each  other;  acquire  the  required 
support;  and  provide  the  required  support 
during  the  Operational  Phase  at  minimum 
cost  (DODI  5000.2,  part  15). 

Integrated  logistic  support  consists  of  10 
specific  components,  or  elements: 


(1)  Maintenance  planning 

(2)  Manpower  and  personnel 

(3)  Supply  support 

(4)  Support  equipment 

(5)  Technical  data 

(6)  Training  and  training  support 

(7)  Computer  resources  support 

(8)  Facilities 

(9)  Packaging,  handling,  storage  and 
transportation 

(10)  Design  interface. 

20.2  PLANNING  FOR  ILS  T&E 

20.2.1  Objectives  of  ILS  T&E 

The  main  objective  of  ILS  T&E  is  to  verify 
that  the  logistic  support  being  developed 
for  the  materiel  system  is  capable  of  meet¬ 
ing  the  required  objectives  for  peacetime 
and  wartime  employment.  The  ILS  T&E 
consists  of  the  usual  development  test  and 
evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E)  but  also  includes 
postdeployment  supportability  assess¬ 
ments.  The  formal  DT&E  and  OT&E  begin 
in  the  Concept  Exploration  and  Definition 
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Phase  and  continue  into  the  Production 
and  Deployment  Phase.  Figure  20-1,  which 
appears  in  the  DSMC  Integrated  Logistics 
Support  Guide,  describes  the  specific  de¬ 
velopment  (DT),  operational  test  (OT)  and 
supportability  assessment  objectives  for 
each  acquisition  phase. 

20.2.2  Planning  Documentation 
for  ILS  T&E 

20.2.2.1  Integrated  Logistic  Support  Plan 

The  ILS  manager  for  a  materiel  acquisition 
system  is  responsible  for  de\  loping  the 
Integrated  Logistic  Support  Plan  (ILSP), 
which  is  the  primary  document  for  plan¬ 
ning  and  implementing  the  support  of  the 
fielded  system.  It  is  initially  prepared  dur¬ 
ing  the  Concept  Exploration  and  Defini¬ 
tion  Phase,  and  progressively  developed  in 
more  detail  as  the  system  moves  through 
the  acquisition  phases.  Identification  of  the 
specific  ILS  test  issues  related  to  the  indi¬ 
vidual  ILS  elements  and  the  overall  system 
support  and  readiness  objectives  are  in¬ 
cluded  in  the  ILSP. 

The  ILS  manager  is  assisted  throughout  the 
system's  development  by  the  Integrated 
Logistics  Support  Management  Team 
(ILSMT),  which  is  formed  early  in  the  ac¬ 
quisition  cycle.  The  ILSMT  is  a  coordina¬ 
tion/advisory  group  composed  of  person¬ 
nel  from  the  program  management  office, 
the  using  command  and  other  commands 
concerned  with  acquisition  activities  such 
as  logistics,  testing  and  training. 

20.2.2.2  Supportability  Assessment  Plan 

Based  upon  the  ILSP  objectives,  the  ILS 
manager,  in  conjunction  with  the  system's 
test  manager,  develops  the  Supportability 
Assessment  Plan  (data  item  description  DI- 
5-7120).  This  plan  identifies  the  testing  ap¬ 
proach  and  the  evaluation  criteria  that  will 


be  used  to  assess  the  supportability-related 
design  requirements  (e.g.,  reliability  and 
maintainability)  and  adequacy  of  the 
planned  logistic  support  resources  for  the 
materiel  system.  Development  of  the  Sup¬ 
portability  Assessment  Plan  begins  in  the 
Concept  Exploration  and  Definition  Phase; 
the  plan  is  then  updated  and  refined  in 
each  successive  acquisition  phase.  The  ILS 
manager  applies  the  techniques  of  logistic 
support  analysis  as  described  in  MIL-STD- 
1388-1  A.  Test  and  evaluation  strategy  is 
formulated,  T&E  program  objectives  and 
criteria  are  established  and  required  test 
resources  are  identified.  The  ILS  manager 
ensures  that  T&E  strategy  is  based  upon 
quantified  supportability  requirements  and 
addresses  supportability  issues  including 
those  with  a  high  degree  of  associated  risk. 
Also,  the  ILS  manager  ensures  that  the 
necessary  quantities  and  types  of  data  will 
be  collected  during  system  development 
and  after  deployment  of  the  system  to  vali¬ 
date  the  various  T&E  objectives.  The  T&E 
objectives  and  criteria  must  provide  a  basis 
that  ensures  critical  supportability  issues 
and  requirements  are  resolved  or  achieved 
within  acceptable  confidence  levels. 

20.2.2.3  Test  and  Evaluation 
Master  Plan  (TEMP) 

The  program  manager  must  include  ILS 
T&E  information  in  the  TEMP  as  specified 
in  DOD  5000.2-M.  The  input,  which  is  de¬ 
rived  from  the  Supportability  Assessment 
Plan  with  the  assistance  of  the  ILS  manager 
and  the  tester,  includes  descriptions  of  re¬ 
quired  operational  suitability,  specific  plans 
for  testing  logistics  supportability  and  re¬ 
quired  testing  resources.  It  is  of  critical 
importance  that  all  key  test  resources  re¬ 
quired  for  ILS  testing  (DT,  OT,  and 
postdeployment  supportability)  be  identi¬ 
fied  in  the  TEMP  because  the  TEMP  pro¬ 
vides  a  long-range  alert  upon  which  test 
resources  are  budgeted  and  obtained  for 
testing. 
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20.2.3  Planning  Guidelines 
for  Logistic  T&E 

The  following  guidelines  for  ILS  T&E  were 
selected  from  those  listed  in  the  DSMC  ILS 
Guide: 

(1)  Develop  a  test  strategy  for  each  ILS- 
related  objective.  Ensure  that  OT&E  plan¬ 
ning  encompasses  all  ILS  elements.  The 
general  ILS  objectives  shown  in  Figure  20- 
1  must  be  translated  into  detailed  quantita¬ 
tive  and  qualitative  requirements  for  each 
acquisition  phase  and  each  T&E  program. 

(2)  Incorporate  ILS  testing  requirements 
(where  feasible)  into  the  formal  DT&E/ 
OT&E  plans. 

(3)  Identify  ILS  T&E  that  will  be  per¬ 
formed  outside  of  the  normal  DT&E  and 
OT &E .  Include  subsystems  that  require  off- 
system  evaluation. 

(4)  Identify  all  required  resources,  includ¬ 
ing  test  articles  and  logistic  support  items 
for  formal  DT /OT  and  separate  ILS  testing 
(participate  with  test  planner). 

(5)  Ensure  establishment  of  an  operation¬ 
ally  realistic  test  environment,  to  include 
personnel  representatives  of  those  who  will 
eventually  operate  and  maintain  the  fielded 
system.  ITiese  personnel  should  be  trained 
for  the  test  using  prototypes  of  the  actual 
training  courses  and  devices.  They  should 
be  supplied  with  drafts  of  all  technical 
manuals  and  documentation  that  will  be 
used  with  the  fielded  system. 

(6)  Ensure  planned  OT&E  will  provide 
sufficient  data  on  high-cost  and  high-main¬ 
tenance  burden  items  (e.g.,  for  high-cost 
critical  spares,  early  test  results  can  be  used 
to  reevaluate  selection). 

(7)  Participate  early  and  effectively  in  the 
TEMP  development  process  to  ensure  the 


TEMP  includes  critical  logistic  T&E  and 
needed  ILS  test  funds  from  program  and 
budget  documents. 

(8)  Identify  the  planned  utilization  of  all 
data  collected  during  the  assessments  to 
avoid  mismatching  of  data  collection  and 
information  requirements. 

Detailed  evaluation  criteria  for  each  of  the 
10  ILS  elements  listed  above  are  presented 
in  Department  of  the  Army  Pamphlet  700- 
50,  "Integrated  Logistic  Support:  Develop¬ 
mental  Supportability  Test  and  Evaluation 
Guide." 

20.3  CONDUCTING  ILS  T&E 
20.3.1  The  Process 

The  purposes  of  ILS  T&E  are  to  measure  the 
supportability  of  a  developing  system 
throughout  the  acquisition  process,  to  iden¬ 
tify  supportability  deficiencies  and  poten¬ 
tial  corrections/improvements  as  test  data 
becomes  available,  and  to  assess  the  opera¬ 
tional  suitability  of  the  planned  support 
system.  The  ILS  T&E  also  evaluates  the 
system's  ability  to  achieve  planned  readi¬ 
ness  objectives  for  the  system/ equipment 
being  developed.  Specific  ILS  T&E  tasks  (as 
prescribed  in  MIL-STD-1388-1A)  include: 

•  Analysis  of  test  results  to  verify  achieve¬ 
ment  of  specified  supportability  require¬ 
ments; 

•  Determination  of  improvements  in  sup¬ 
portability  and  supportability-related  de¬ 
sign  parameters  needed  for  the  system  to 
meet  established  goals  and  thresholds; 

•  Identification  of  areas  where  estab¬ 
lished  goals  and  thresholds  have  not  been 
demonstrated  within  acceptable  confidence 
levels; 
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•  Development  of  corrections  for  identi¬ 
fied  supportability  problems  such  as  modi¬ 
fications  to  hardware,  software,  support 
plans,  logistic  support  resources  or  opera¬ 
tional  tactics; 

•  Projection  of  changes  in  costs,  readi¬ 
ness  and  logistic  support  resources  due  to 
implementation  of  corrections; 

•  Analysis  of  supportability  data  from 
the  deployed  system  to  verify  achievement 
of  the  established  goals  and  thresholds  and 
where  operational  results  deviate  from  pro¬ 
jections,  determination  of  the  causes  and 
corrective  actions. 

Integrated  logistics  support  T&E  may  con¬ 
sist  of  a  series  of  ILS  demonstrations  and 
assessments  that  are  usually  conducted  as 
part  of  system  performance  tests.  Special 
end-item  equipment  tests  are  rarely  con¬ 
ducted  solely  for  ILS  evaluation. 

20.3.2  Reliability  and  Maintainability 

System  availability  is  generally  considered 
to  be  compoc  ed  of  two  major  system  char¬ 
acteristics  —  reliability  and  maintainabil¬ 
ity.  The  DODI  5000.2  states: 

Reliability  (R)  is  the  ability  of  a  system  and 
its  parts  to  perform  its  mission  without 
failure,  degradation,  or  demand  on  the  sup¬ 
port  system. 

Maintainability  (Ml  is  the  ability  of  an  item 
to  be  retained  in  or  restored  to  specified 
condition  when  maintenance  is  performed 
by  personnel  having  specific  skill  levels, 
using  prescribed  procedures  and  resources, 
at  each  prescribed  level  of  maintenance 
and  repair. 

Operational  Reliability  and  Maintainabil¬ 
ity  Value  is  any  measure  of  reliability  or 
maintainability  that  includes  the  combined 


effects  of  item  design,  quality,  installation, 
environment,  operation,  maintenance,  and 
repair. 

The  R  and  M  program  objectives  are  to  be 
defined  as  system  parameters  early  in  the 
development  process.  They  will  be  used  as 
evaluation  criteria  throughout  the  design, 
development  and  production  processes. 
"Reliability  and  maintainability  objectives 
will  be  translated  into  quantifiable  contrac¬ 
tual  terms  and  allocated  through  the  sys¬ 
tem  design  hierarchy."  An  understanding 
of  how  this  allocation  affects  testing  oper¬ 
ating  characteristics  below  system  level  can 
be  found  in  DOD  3235.1-H,  "T&E  of  Sys¬ 
tem  Reliability,  Availability  and  Maintain¬ 
ability."  This  is  especially  important  to  test¬ 
ing  organizations  expected  to  make  early 
predictions  of  system  performance.  Guid¬ 
ance  on  testing  reliability  may  also  be  found 
in  MIL-STD-78I,  "Reliability  Testing  for 
Engineering  Development,  Qualification, 
and  Production." 

20.3.2.1  Reliability 

In  MIL-STD-785,  the  following  is  discussed: 

Environmental  Stress  Screening  (ESSl  is  a 
test,  or  series  of  tests  during  engineering 
development,  specifically  designed  to  iden¬ 
tify  weak  parts  or  manufacturing  defects. 
Test  conditions  should  stimulate  failures 
typical  of  early  field  service  rather  than 
provide  an  operational  life  profile. 

Reliability  Development /Growth  Testing 
(RDT/RGTl  is  a  systematic  engineering 
process  of  test-analyze-fix-retest  (TAFT) 
where  equipment  is  tested  under  actual, 
simulated,  or  accelerated  environments.  It 
is  an  iterative  methodology  intended  to 
rapidly  and  steadily  improve  reliability. 

Reliability  Qualification  Test  fROTl  is  to 
verify  that  minimum  acceptable  reliability 
requirements  have  been  met  before  items 
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are  committed  to  production.  A  statistical 
test  plan  is  used  to  predefine  criteria  which 
will  limit  government  risk.  Test  conditions 
must  be  operationally  realistic. 

Production  Reliability  Acceptance  Test 
(PRAT)  is  intended  to  simulate  in-service 
use  of  the  delivered  item  or  production  lot. 
"Because  it  must  provide  a  basis  for  deter¬ 
mining  contractual  compliance,  and  be¬ 
cause  it  applies  to  the  items  actually  deliv¬ 
ered  to  operational  forces,  PRAT  must  be 
independent  of  the  supplier  if  at  all  pos¬ 
sible".  PRAT  may  require  expensive  test 
facilities,  so  100%  sampling  is  not  recom¬ 
mended. 

20.3.2.2  Maintainability 

Maintainability  design  factors  and  test/ 
demonstration  requirements  used  to  evalu¬ 
ate  maintainability  characteristics  must  be 
based  on  program  objectives  and  thresh¬ 
olds.  Areas  for  evaluation  might  include 
(DOD  3235.1-H): 

Accessibility:  Assess  how  easily  the 
item  can  be  repaired  or  adjusted. 

Visibility:  Assess  the  ability /need  to 
see  the  item  being  repaired. 

Testability:  Assess  ability  to  detect  and 
isolate  system  faults  to  the  faulty  re¬ 
placeable  assembly  level. 

Complexity:  Assess  the  impact  of  the 
number,  location  and  characteristic 
(standard  or  special  purpose)  on  sys¬ 
tem  maintenance. 

Interchangeability:  Assess  the  level  of 
difficulty  encountered  when  failed  or 
malfunctioning  parts  are  removed  or 
replaced  with  an  identical  part  not 
requiring  recalibration. 


"A  true  assessment  of  system  maintain¬ 
ability  generally  must  be  developed  at  the 
system  level  under  operating  conditions 
and  using  production  configuration  hard¬ 
ware."  Therefore,  DODI  5000.2,  part  6-C, 
requires  that  a  maintainability  demonstra¬ 
tion  (MIL-STD-470)  be  conducted  prior  to 
Milestone  III. 

20.3.3  T&E  of  System  Support  Package 

The  T&E  of  the  support  for  a  materiel  sys¬ 
tem  requires  a  system  support  package 
consisting  of  spares,  support  equipment, 
technical  documents  and  publications,  rep¬ 
resentative  personnel,  any  peculiar  sup¬ 
port  requirements  and  the  test  article  itself, 
in  short,  all  of  the  items  that  would  eventu¬ 
ally  be  required  when  the  system  is  opera¬ 
tional.  This  complete  support  package  must 
be  at  the  test  site  before  the  test  is  scheduled 
to  begin.  Delays  in  the  availability  of  cer¬ 
tain  support  items  could  prevent  the  test 
from  proceeding  on  schedule.  This  could 
be  costly  due  to  on-site  support  personnel 
on  hold  or  tightly  scheduled  system  ranges 
and  expensive  test  resources  not  being  prop¬ 
erly  utilized.  Also,  it  could  result  in  the  test 
proceeding  without  conducting  the  com¬ 
plete  evaluation  of  the  support  system.  The 
ILS  test  planner  must  ensure  that  the  re¬ 
quired  personnel  are  trained  and  available, 
the  test  facility  scheduling  is  flexible  enough 
to  permit  normal  delays,  and  the  test  sup¬ 
port  package  is  on  site  on  time. 

20.3.4  Data  Collection  and  Analysis 

The  ILS  manager  must  coordinate  with  the 
testers  to  ensure  that  the  methods  used  for 
collection,  storage  and  extraction  of  ILS 
T&E  data  are  compatible  with  those  used  in 
testing  the  materiel  system.  As  with  any 
testing,  the  ILS  test  planning  must  ensure 
that  all  required  data  is  identified;  it  is 
sufficient  to  evaluate  a  system's  readiness 
and  supportability;  and  plans  are  made  for 


20-6 


a  data-manageme  >t  system  that  is  capable 
of  the  data  classification,  storage,  retrieval 
and  reduction  necessary  for  statistical  analy¬ 
sis.  Large  statistical  sample  sizes  may  re¬ 
quire  a  common  database  that  integrates 
contractor,  DT&E  and  OT&E  data  so  re¬ 
quired  performance  parameters  can  be 
demonstrated. 

20.3.5  Use  of  ILS  Test  Results 

The  emphasis  on  the  use  of  the  results  of 
testing  changes  as  the  program  moves  from 
the  CED  Phase  to  postdeployment.  During 
early  phases  of  a  program,  the  evaluation 
results  are  used  primarily  to  verify  analysis 
and  develop  future  projections.  As  the  pro¬ 
gram  moves  into  EMD  and  hardware  be¬ 
comes  available,  the  evaluation  addresses 
design,  particularly  the  reliability  and  main¬ 
tainability  aspects;  training  programs;  sup¬ 
port  equipment  adequacy;  personnel  skills 
and  availability;  and  technical  publications. 

The  ILS  manager  must  make  the  program 
manager  (PM)  aware  of  the  impact  on  the 
program  of  logistical  shortcomings  that  are 
identified  during  the  T&E  process.  The  PM, 
in  turn,  must  ensure  that  the  solutions  to 
any  shortcomings  are  identified  and  re¬ 
flected  in  the  revised  specifications  and 
that  the  revised  test  requirements  are  in¬ 
cluded  in  the  updated  TEMP  as  the  pro¬ 
gram  proceeds  through  the  various  acquisi¬ 
tion  stages. 

20.4  LIMITATIONS  TO  ILS  T&E 

Concurrent  testing  or  tests  that  have  accel¬ 
erated  schedules  frequently  do  not  have 
sufficient  test  articles,  equipment  or  hard¬ 
ware  to  achieve  statistical  confidence  in  the 


testing  conducted.  Tlie  shortage  of  equip¬ 
ment  is  often  the  reason  that  shelf-life  and 
service-life  testing  is  incomplete,  leaving 
the  ILS  evaluator  with  insufficient  data  to 
predict  future  performance  of  the  test  item. 
Some  evaluations  must  measure  perfor¬ 
mance  against  a  point  on  the  parameter's 
growth  curve.  The  ILS  testing  will  continue 
postproduction  to  obtain  required  sample 
sizes  for  verifying  performance  criteria. 
Many  aspects  of  the  logistic  support  sys¬ 
tem  may  not  be  available  for  lOT&E  and 
become  testing  limitations.  The  PMO  must 
develop  enough  logistic  support  to  ensure 
the  user  can  maintain  the  system  during 
lOT&E  without  requiring  system  contrac¬ 
tor  involvement.  Any  ILS  limitations  upon 
lOT&E  will  likely  be  evaluated  during 
FOT&E. 

20.5  SUMMARY 

Test  and  evaluation  are  the  logisticians' 
tools  for  measuring  the  ability  of  the 
planned  support  system  to  fulfill  the  mate¬ 
riel  system's  readiness  and  supportability 
objectives.  The  effectiveness  of  ILS  T&E  is 
based  upon  the  completeness  and  timeli¬ 
ness  of  the  planning  effort. 

The  ILS  T&E  requirements  must  be  an  inte¬ 
gral  part  of  the  TEMP  to  ensure  budgeting 
and  scheduling  of  required  test  resources. 
Data  requirements  must  be  completely 
identified,  with  adequate  plans  made  for 
collection,  storage,  retrieval  and  reduction 
of  test  data.  At  MS  II,  decision-makers  can 
expect  that  some  ILS  performance  param¬ 
eters  will  not  have  finished  testing  because 
of  the  large  sample  sizes  required  for  statis¬ 
tical  analysis. 
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EC/C3  TEST  AND  EVALUATION 


21.1  INTRODUCTION 

Testing  of  electronic  combat  (EC)  and  com¬ 
mand,  control  and  communications  (C3) 
systems  pose  unique  problems  for  the  tester 
because  of  the  difficulty  in  measuring  their 
operational  performance.  Special  testing 
techniques  and  facilities  are  normally  re¬ 
quired  in  EC  and  C3  testing.  This  chapter 
discusses  the  problems  associated  with  EC 
and  C3  testing  and  presents  methodologies 
the  tester  can  consider  using  to  overcome 
the  problems. 

21.2  TESTING  EC  SYSTEMS 

21.2.1  Special  Consideration  When 
Testing  EC  Systems 

Electronic  combat  systems  operate  across 
the  electromagnetic  spectrum,  performing 
offensive  and  defensive  support  roles.  Con¬ 
figurations  vary  from  subsystem  compo¬ 
nents  to  full-up  independent  systems.  The 
EC  systems  are  used  to  increase  survivabil¬ 
ity,  degrade  enemy  capability  and  contrib¬ 
ute  to  the  overall  success  of  the  combat 
mission.  Decision-makers  want  to  know 
the  incremental  contribution  to  total  force 
effectiveness  made  by  a  new  EC  system 
when  measured  in  a  force-on-force  engage¬ 
ment.  However,  the  contractual  specifica¬ 
tions  for  EC  systems  are  usually  stated  in 
terms  of  engineering  parameters  such  as 
effective  radiated  power,  reduction  in  com- 
mimications  intelligibility  and  jamming- 
to-signal  ratio.  These  measures  are  of  little 


use  by  themselves  in  assessing  contribu¬ 
tion  to  mission  success.  The  decision-mak¬ 
ers  require  that  testing  be  conducted  under 
realistic  operational  conditions;  but  the 
major  field  test  ranges,  such  as  the  shore¬ 
line  at  Eglin  AFB  or  the  desert  at  Nellis 
AFB,  cannot  provide  the  signal  density  or 
realism  of  threats  that  would  be  presented 
by  regional  conflicts  in  central  ’"urope.  In 
field  testing,  the  tester  can  achieve  one-on- 
one  or,  at  best,  few-on-few  testing  condi- 
tioi\s.  To  do  this  he  needs  a  methodology 
that  will  permit  extrapolation  of  engineer¬ 
ing  measurements  and  one-on-one  test 
events  to  create  more  operationally  mean¬ 
ingful  measures  of  mission  success  in  a 
force-on-force  context,  usually  under  simu¬ 
lated  conditions. 

21.2.2  Integrated  Test  Approach 

An  integrated  approach  to  EC  testing  using 
a  combination  of  large-scale  models,  com¬ 
puter  simulations,  hybrid  man-in-the-loop 
simulators  and  field  test  ranges  is  a  solu¬ 
tion  for  the  EC  tester.  No  tool  by  itself  is 
adequate  to  provide  a  comprehensive 
evaluation.  Simulation,  both  digital  and 
hybrid,  can  provide  a  means  for  efficient 
test  execution.  Computer  models  can  be 
used  to  simulate  many  different  test  cases 
to  aid  the  tester  in  assessing  the  critical  test 
issues  (i.e.,  sensitivity  analysis)  and  pro¬ 
duce  a  comprehensive  set  of  predicted  re¬ 
sults.  As  digital  simulation  models  are  vali- 


dated  with  empirical  data  from  testing, 
they  can  be  used  to  evaluate  the  system 
under  test  in  a  more  dense  and  complex 
threat  environment  and  at  expected  war¬ 
time  levels.  In  addition,  the  field  test  results 
are  used  to  validate  the  model;  and  the 
model  is  used  to  validate  the  field  tests, 
thus  lending  more  credibility  to  both  re¬ 
sults.  Hybrid  man-in-the-loop  simulators, 
such  as  the  Real-Time  Electromagnetic  Digi¬ 
tally  Controlled  Analyzer  and  Processor 
(REEXZAP)  and  the  Air  Force  Electronic 
Warfare  Evaluation  Simulator  (AFEWES) 
can  provide  a  capability  to  test  against  new 
threats.  Hybrid  simulators  cost  less  and  are 
safer  than  field  testing.  The  field  test  ranges 
are  used  when  a  wider  range  of  actions  and 
reactions  by  aircraft  and  ground  threat  sys¬ 
tem  operations  is  required. 

Where  one  tool  is  weak,  another  may  be 
strong.  By  using  all  the  tools,  an  EC  tester 
can  do  a  complete  job  of  testing.  An  ex¬ 
ample  of  an  integrated  methodology  is 
shown  in  Figure  21-1.  The  EC  integrated 
testing  can  be  summarized  as: 

(1)  Initial  modeling  phase  for  sensitivity 
analysis  and  test  planning, 

(2)  Active  test  phases  at  hybrid  labora¬ 
tory  simulator  and  field  range  facilities, 

(3)  Test  data  reduction  and  analysis, 

(4)  Post-test  modeling  phase  repeating 
the  first  step  using  test  data  for  extrapola¬ 
tion, 

(5)  Force  effectiveness  modeling  and 
analysis  phase  to  determine  the  incremen¬ 
tal  contribution  of  the  new  system  to  total 
force  effectiveness. 

Another  alternative  is  the  electronic  com¬ 
bat  test  process  proposed  in  the  "Air  Force 
Electronic  Combat  Development  Test  Pro¬ 


cess  Guide,"  May  1991,  issued  by  what  is 
now  the  Air  Staff  T&E  Element,  AF/TE. 
The  six  step  process  described  here  is 
graphically  represented  by  Figure  21-2:. 

(1)  Deriving  test  requirements, 

(2)  Conducting  pretest  analysis  to  pre¬ 
dict  EC  system  performance, 

(3)  Conducting  test  sequences  under  pro¬ 
gressively  more  rigorous  ground-  and 
flight-test  conditions, 

(4)  Processing  test  data, 

(5)  Conducting  post-test  analysis  and 
evaluation  of  operational  effectiveness  and 
suitability, 

(6)  Feeding  results  back  to  the  system; 
development  employment  process. 

As  can  be  seen  from  Figure  21-3,  assuming 
a  limited  budget  and  field  test  being  the 
most  expensive  per  number  of  trials,  the 
cost  of  test  trials  forces  the  developer  and 
tester  to  make  trade-offs  to  obtain  the  neces¬ 
sary  test  data.  Many  more  iterations  of  a 
computer  simulation  can  be  run  for  the  cost 
of  an  open-air  test. 

21.3  TESTING  OF  C3  SYSTEMS 

21.3.1  Special  Considerations 
When  Testing  C3  Systems 

The  purpose  of  a  C3  system  is  to  provide  a 
commander  with  timely  and  relevant  in¬ 
formation  to  support  sound  decision-mak¬ 
ing.  A  variety  of  problems  face  the  C3  sys¬ 
tem  tester.  However,  in  evaluating  com¬ 
mand  effectiveness,  it  is  difficult  to  sepa¬ 
rate  the  contribution  made  by  the  C3  sys¬ 
tem  from  the  contribution  made  by  the 
commander's  innate,  cognitive  processes. 
To  assess  a  C3  system  in  its  operational 
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Figure  21  -1 .  Integrated  EC  Testing  Approach 
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environment  it  must  be  connected  to  the 
other  systems  with  which  it  would  nor¬ 
mally  operate,  making  traceability  of  test 
results  difficult.  Additionally,  modem  C3 
systems  are  software  intensive  and  highly 
interactive,  with  complex  man-machine 
interfaces.  Measuring  C3  system  effective¬ 
ness  thus  requires  the  tester  to  use  properly 
trained  user  troops  during  the  test  and  to 
closely  monitor  software  test  and  evalua¬ 
tion  (T&E).  The  C3  systems  of  the  Army, 
Navy,  Air  Force  and  Marines  are  expected 
to  interoperate  with  each  other  and  with 
those  of  the  NATO  forces;  hence,  the  tester 
must  also  ensure  inter-Service  and  NATO 
compatibility  and  interoperability. 

21.3.2  C3  Test  Facilities 

Testing  of  C3  systems  will  have  to  rely 
more  on  the  use  of  computer  simulations 
and  command,  control,  communication, 
intelligence  (C3I)  test-beds  to  assess  their 
overall  effectiveness.  The  Joint  Tactical 
Command,  Control,  and  Communications 
Agency  GTC3A),  which  is  responsible  for 
ensuring  interoperability  among  all  U.S. 
tactical  C3  systems  that  would  be  used  in 
joint  or  combined  operations,  operates  the 
Joint  Interoperability  Test  Center  in  Ft. 
Huachuca,  Ariz.  The  center  is  a  test-bed  for 
C3I  systems  interoperability.  Another  fa¬ 
cility,  the  huge  test-bed  developed  at 
Kirtland  AFB,  N.M.,  for  the  Identification 
Friend,  Foe  or  Neutral  (IFFN)  Joint  Test, 
will  be  operated  by  the  Air  Force  and  be 
available  for  use  by  the  development  and 
operational  communities  of  all  the  Services 
for  their  C3I  testing  needs. 

21.4  TRENDS  IN  TESTING  C3  SYSTEMS 

21.4.1  Evolutionary  Acquisition 
of  C3  Systems 

Evolutionary  Acquisition  (E  A)  is  a  strategy 
designed  to  provide  an  early,  useful  ca¬ 


pability  even  though  detailed  overall  sys¬ 
tem  requirements  cannot  be  fully  defined 
at  the  program's  inception.  The  EA  strat¬ 
egy  contributes  to  a  reduction  in  the  risks 
involved  in  system  acquisition,  since  the 
system  is  developed  and  tested  in  manage¬ 
able  increments.  The  C3  systems  are  likely 
candidates  for  E  A  because  they  are  charac¬ 
terized  by  system  requirements  that  are 
difficult  to  quantify  or  even  articulate  and 
that  are  expected  to  change  as  a  function  of 
scenario,  mission,  theater,  threat  and  emerg¬ 
ing  technology.  Therefore,  the  risk  associ¬ 
ated  with  developing  these  systems  can  be 
very  great. 

Studies  by  the  Defense  Systems  Manage¬ 
ment  College  and  the  International  Test 
and  Evaluation  Association  (ITEA)  have 
addressed  the  issues  involved  in  the  evolu¬ 
tionary  acquisition  and  testing  of  C3  sys¬ 
tems.  The  ITEA  study  illustrated  EA  in 
Figure  21-4  and  stated  that: 

With  regard  to  the  tester's  role  in  EA, 
the  study  group  concluded  that  itera¬ 
tive  test  and  evaluation  is  essential  for 
success  in  an  evolutionary  acquisition. 
The  tester  must  become  involved  early 
in  the  acquisition  process  and  contrib¬ 
ute  throughout  the  development  and 
fielding  of  the  core  and  the  subsequent 
increments. ...The  testers  contribute  to 
the  requirements  process  through  feed- 
back  of  test  results  to  the 
user.. .and. ..must  judge  the  ability  of 
the  system  to  evolve.  [Reference  4] 

The  testing  of  EA  systems  presents  the 
tester  with  a  unique  challenge  as  the  core 
system  must  be  tested  during  fielding  and 
the  first  increment  before  the  core  testing  is 
completed.  This  could  lead  to  a  situation 
where  the  tester  has  three  or  four  tests 
ongoing  on  various  increments  of  the  same 
system.  The  program  manager  must  insist 
that  the  testing  for  EA  systems  be  carefully 
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planned  to  ensure  the  test  data  is  shared  by 
all  and  there  is  a  minimum  of  repetition  or 
duplication  in  testing. 

21.4.2  Radio  Vulnerability 

The  Radio  Vulnerability  Analysis  (RVAN) 
methodology  is  for  assessing  the  anti-jam 
capability  and  limitations  of  radio  fre¬ 
quency  data  links  when  operating  in  a  hos¬ 
tile  electronic  countermeasures  environ¬ 
ment.  The  RVAN  evolved  from  the  test 
methodologies  developed  for  an  Office  of 
the  Secretary  of  Defense  (OSD)-chartered 
Joint  Test  on  Data  Link  Vulnerability  Analy¬ 
sis  (DVAL).  In  1983,  OSD  directed  the  Ser¬ 
vices  to  apply  the  DVAL  methodology  to 
all  new  data  links  being  developed. 

The  purpose  of  the  DVAL  methodology  is 
to  identify  and  quantify  the  antijam  capa¬ 
bilities  and  vulnerabilities  of  a  radio  fre¬ 
quency  (RF)  data  link  operating  in  a  hostile 
electronic  countermeasures  (ECM)  envi¬ 
ronment.  The  methodology  is  applied 
throughout  the  acquisition  process  and  per¬ 
mits  early  identification  of  needed  design 
modifications  to  reduce  identified  ECM 
vulnerabilities.  The  following  four  compo¬ 
nents  determine  a  data  link's  electronic 
warfare  (EW)  vulnerability: 

fl)  The  susceptibility  of  a  data  link;  i.e., 
the  receiver's  performance  when  subjected 
to  intentional  threat  ECM; 

(2)  The  interceptibility  of  the  data  link; 
i.e.,  the  degree  to  which  the  transmitter 


could  be  intercepted  by  enemy  intercept 
equipment; 

(3)  The  accessibility  of  the  data  link;  i.e., 
the  likelihood  that  a  threat  jammer  could 
degrade  the  data  imk's  performance; 

(4)  The  feasibility  that  the  enemy  would 
intercept  and  jam  the  data  link  and  success¬ 
fully  degrade  its  performance. 

The  analyst  applying  the  DVAL  methodol¬ 
ogy  will  require  test  data;  and  the  test  man¬ 
ager  of  the  C3I  system,  of  which  the  data 
link  is  a  component,  will  be  required  to 
provide  this  data.  The  DVAL  joint  test 
methodologies  and  test  results  are  on  file  as 
part  of  the  Joint  Test  Library  being  main¬ 
tained  by  the  USAF  Operational  Test  and 
Evaluation  Center,  Kirtland  AFB,  N.M. 

21.5  SUMMARY 

The  EC  systems  must  be  tested  under  con¬ 
ditions  representative  of  the  dense  threat 
signal  environments  in  which  they  will 
operate.  The  C3  systems  must  be  tested  in 
representative  environments  where  their 
interaction  and  responsiveness  can  be  dem¬ 
onstrated.  The  solution  for  the  tester  is  an 
integrated  approach  using  a  combination 
of  analytical  models,  computer  simulations, 
hybrid  laboratory  simulators  and  test  beds, 
and  actual  field  testing.  The  tester  must 
understand  these  test  techniques  and  re¬ 
sources  and  apply  them  in  EC  and  C3  test 
and  evaluation. 
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22 

MULTI-SERVICE  TESTS 


22.1  INTRODUCTION 

This  chapter  discusses  the  planning  and 
management  of  a  multi-Service  test  pro¬ 
gram.  A  multi-Service  test  program  is  con¬ 
ducted  when  a  system  is  to  be  acquired  for 
use  by  more  than  one  Service  or  when  a 
system  must  interface  with  equipment  of 
another  Service.  A  multi-Service  test  pro¬ 
gram  should  not  be  confused  with  the  (DSD- 
sponsored,  nonacquisition-oriented  Joint 
Test  and  Evaluation  (JT&E)  program.  A 
brief  description  of  the  JT&E  program  is 
provided  in  Chapter  6. 

22.2  BACKGROUND 

A  definition  of  multi-Service  test  and  evalu¬ 
ation  is  contained  in  EXDDI  5000.2.  It  desig¬ 
nates  the  participants  in  the  program  and 
gives  a  Lead  Service  responsibility  for  pre¬ 
paring  a  single  report  concerning  a  system's 
operational  effectiveness  and  suitability. 
(The  Lead  Service  is  the  Service  responsible 
for  the  overall  management  of  a  multi- 
Service  program.  A  "Supporting  Service" 
is  a  Service  designated  to  assist  the  Lead 
Service.) 

A  multi-Service  test  and  evaluation  (T&E) 
program  may  include  either  development 
test  and  evaluation  (DT&E)  or  operational 
test  and  evaluation  (OT&E)  or  both.  The 
Service's  operational  test  agencies  have 
executed  a  formal  Memorandum  of  Agree¬ 
ment  on  multi-Service  OT&E  (Reference 
35)  that  provides  a  framework  for  the  con¬ 


duct  of  a  multi-Service  operational  test  pro¬ 
gram. 

Air  Force  Regulation  80-14  describes  the 
procedures  followed  in  a  multi-Service  T &E 
program  as  follows: 

(1)  In  a  multiservice  acquisition  pro¬ 
gram,  T&E  is  planned  and  conducted 
according  to  Lead  Service  regulahons. 
The  designated  Lead  Service  will  have 
the  overall  responsibility  for  manage¬ 
ment  of  the  multi-Service  program  and 
will  ensure  that  supporting  service 
requirements  are  included.  If  another 
Service  has  certain  unique  T&E  re¬ 
quirements,  testing  for  these  unique 
requirements  may  be  planned,  funded, 
and  conducted  according  to  that 
Service's  regulations. 

(2)  Participating  Services  will  prepare 
reports  in  accordance  with  their  re¬ 
spective  regulations.  The  Lead  Service 
will  prepare  and  coordinate  a  single 
DT&E  report  and  a  single  OT&E  re¬ 
port,  which  will  summarize  the  con¬ 
clusions  and  recommendations  of  each 
Service's  reports.  Rationale  willbepro- 
vided  to  explain  any  significant  differ¬ 
ences.  The  individual  Service  reports 
will  be  attached  to  this  single  report. 

(3)  Deviations  from  the  Lead  Service 
T&E  regulations  may  be  accommo- 
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dated  by  mutual  agreement  among 
the  Services  involved. 

22.3  TEST  PROGRAM 
RESPONSIBILITIES 

The  Lead  Service  has  overall  management 
responsibility  for  the  program.  It  must  en¬ 
sure  that  supporting  ^rvice  requirements 
are  included  in  the  formulation  of  the  basic 
resource  and  planning  documents. 

A  Test  Management  Council  (TMC)  should 
be  established  for  each  multi-Service  test 
program.  Its  membership  consists  of  one 
senior  representative  from  each  participat¬ 
ing  Service  or  agency  headquarters.  The 
TMC  works  closely  with  the  program  man¬ 
agement  office  (PMO)  and  is  responsible 
for  arbitrating  all  disagreements  among 
Services  that  cannot  be  resolved  at  the  work¬ 
ing  level. 

Resource  requirements  are  documented  in 
the  Test  and  Evaluation  Master  Plan 
(TEMP).  Each  participating  Service  is  di¬ 
rected  to  budget  for  the  testing  necessary  to 
accomplish  its  assigned  test  objectives  and 
for  the  participation  of  its  personnel  and 
equipment  in  the  entire  test  program.  Sepa¬ 
rate  annexes  may  be  used  to  address  each 
Service's  test  requirements. 

22.4  TEST  TEAM  STRUCTURE 

A  sample  test  team  structure  is  shown  in 
Figure  22-1.  As  shown  in  the  figure.  Service 
test  teams  work  through  a  Service  deputy 
test  director  or  senior  representative.  The 
test  director  exercises  test  management 
authority  but  not  operational  control  over 
the  test  teams.  The  responsibilities  include 
integration  of  test  requirements  and  effi¬ 
cient  scheduling  of  test  events.  The  deputy 
test  directors  exercise  operational  control 
or  test  management  authority  over  their 
Service  test  teams  in  accordance  with  their 


Service  directives.  Additionally,  they  act  as 
advisers  to  the  test  director;  represent  their 
Service's  interests;  and  are  responsible,  at 
least  administratively,  for  resources  and 
personnel  provided  by  their  Services. 

22.5  TEST  PLANNING 

Test  planning  for  multi-Service  T&E  is  ac¬ 
complished  in  the  manner  prescribed  by 
Lead  Service  directions  and  in  accordance 
with  the  following  general  procedures  ex¬ 
tracted  from  the  "Memorandum  of  Agree¬ 
ment  on  Multi-Service  OT&E  and  Joint 
T&E": 

(1)  The  Lead  Service  T&E  agency  begins 
the  planning  process  by  issuing  a  call  to 
the  supporting  Service  T&E  agencies  for 
critical  issues  and  test  objectives. 

(2)  The  Lead  Service  T&E  agency  con¬ 
solidates  the  objectives  into  a  list  and 
coordinates  the  list  with  the  supporting 
Service  T&E  agencies. 

(3)  The  Lead  Service  T&E  agency  accom¬ 
modates  supporting  Service  T&E  require¬ 
ments  and  input  in  the  formal  coordina¬ 
tion  action  of  the  TEMP. 

(4)  Participating  T&E  agency  project  of¬ 
ficers  assign  responsibility  for  the  ac¬ 
complishment  of  test  objectives  (from 
the  consolidated  list)  to  each  T&E  agency. 
These  assignments  are  made  in  a  mutu¬ 
ally  agreeable  manner.  Each  agency  is 
then  responsible  for  resource  identifica¬ 
tion  and  accomplishment  of  its  assigned 
test  objectives  under  the  direction  of  the 
Lead  Service  T&E  agency. 

(5)  Each  participating  agency  prepares 
the  portion  of  the  overall  test  plan(s)  for 
its  assigned  objectives,  in  the  Lead  Ser¬ 
vice  test  plan(s)  format,  and  identifies  its 
data  needs. 
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(6)  The  Lead  Service  T&E  agency  pre¬ 
pares  the  multi-Service  T&E  test  plan(s), 
consolidating  the  input  from  all  partici¬ 
pating  agencies. 

22.6  DISCREPANCY  REPORTING 

In  a  multi-Service  T&E  program,  a  discrep¬ 
ancy  report  is  a  report  of  any  condition  that 
reflects  adversely  on  the  item  being  tested 
and  that  must  be  reported  outside  the  test 
team  for  corrective  action.  The  discrepancy 
reporting  system  of  the  Lead  Service  is 
normally  used.  All  members  of  the  multi- 
Service  test  team  will  report  discrepancies 
through  their  Service's  system. 

Items  undergoing  test  will  not  necessarily 
be  used  by  each  of  the  Services  for  identical 
purposes.  As  a  result,  a  discrepancy  con¬ 
sidered  disqualifying  by  one  Service  is  not 
necessarily  disqualifying  for  all  Services. 
Discrepancy  reports  of  a  disqualifying  na¬ 
ture  must  include  a  statement  by  the  con¬ 
cerned  Service  of  why  the  discrepancy  has 
been  so  classified.  It  also  includes  state¬ 
ments  by  the  other  Services  as  to  whether 
or  not  the  discrepancy  affects  them  signifi¬ 
cantly. 

If  one  of  the  participating  Services  identi¬ 
fies  a  discrepancy  that  it  considers  as  war¬ 
ranting  termination  of  the  test,  the  circum¬ 
stances  are  reported  immediately  to  the 
test  director. 

22.7  TEST  REPORTING 

The  following  test-reporting  policy  applies 
to  multi-Service  OT&E  programs: 

(1)  Interim  test  reports  are  not  normally 
prepared.  If  they  are  required  on  a  par¬ 
ticular  program;  they  are  prepared  in 
accordance  with  Lead  Service  directives 
and  coordinated  with  all  participating 


OT&E  agencies  prior  to  release. 

(2)  Within  60  days  of  the  end  of  testing, 
the  multi-Service  OT&E  team  must 
present  a  factual  report  of  the  test  to  all 
participating  OT&E  agencies.  (This  fac¬ 
tual  report  presents  the  data  collected 
but  no  evaluation,  conclusions  or  recom¬ 
mendations  concerning  the  data.) 

(3)  Each  participating  OT&E  agency  pre¬ 
pares  an  independent  evaluation  report 
in  its  own  format  and  forwards  that  re¬ 
port  through  its  normal  Service  chan¬ 
nels. 

(4)  Approved  independent  evaluation 
reports  are  distributed  to  all  participat¬ 
ing  OT&E  agencies. 

(5)  The  Lead  Service  OT&E  agency  is 
responsible  for  preparing  the  Defense 
Acquisition  Board  (DAB)  briefing(s) 
which  is  (are)  coordinated  with  all  par¬ 
ticipating  OT&E  agencies. 

22.8  SUMMARY 

Multi-Service  test  programs  are  conducted 
by  two  or  more  Services  when  a  system  is  to 
be  acquired  by  more  than  one  Service  or 
when  a  system  must  interface  with  equip¬ 
ment  of  another  Service.  Test  procedures 
for  multi-Service  T&E  follow  those  of  the 
designated  Lead  Service,  with  mutual 
agreements  resolving  areas  where  devia¬ 
tions  are  necessary.  Care  must  be  exercised 
when  integrating  test  results  and  reporting 
discrepancies  since  items  undergoing  test¬ 
ing  may  be  used  for  different  purposes  in 
different  Services.  Close  coordination  is 
required  to  ensure  that  an  accurate  sum¬ 
mary  of  the  developing  system's  capabili¬ 
ties  is  provided  to  Service  and  DOD  deci¬ 
sion  authorities. 
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23 

INTERNATIONAL  TEST 
AND  EVALUATION  PROGRAMS 


23.1  INTRODUCTION 

This  chapter  discusses  test  and  evaluation 
(T&E)  from  an  international  perspective.  It 
describes  the  Office  of  the  Secretary  of 
Defense  (OSD)-sponsored  Foreign  Com¬ 
parative  Test  Program.  Factors  that  bear 
on  the  T&E  of  multinational  acquisition 
programs  are  discussed  also. 

23.2  FOREIGN  COMPARATIVE 
TEST  PROGRAM 

23.2.1  Program  Objective 

The  Foreign  Comparative  Test  (FCT)  Pro¬ 
gram  is  designed  to  support  the  evaluation 
of  a  foreign  nation's  weapons  system, 
equipment  or  technology  in  terms  of  its 
potential  to  meet  a  valid  requirement  of 
one  or  more  of  the  U.S.  Armed  Services. 
Additional  goals  of  the  FCT  program  in¬ 
clude  avoiding  unnecessary  duplication  in 
development,  enhancing  standardization 
and  interoperability  and  promoting  inter¬ 
national  technology  exchanges.  The  FCT 
program  is  not  intended  for  use  in  exploit¬ 
ing  threat  systems  or  for  intelligence  gath¬ 
ering.  The  primary  objective  of  the  pro¬ 
gram  is  to  reduce  the  costs  of  research  and 
development,  while  leading  to  the  acquisi¬ 
tion  of  foreign  equipment  for  U.S.  use. 
Policy  and  procedures  for  the  execution  of 
the  program  are  documented  in  DOD 
5000.3-M-2. 


23.2.2  Program  Administration 

Foreign  weapons  evaluation  activities  and 
responsibilities  are  assigned  to  the  Direc¬ 
tor,  Test  and  Evaluation  (DTE)  by  direction 
of  the  Congress  in  1980.  Each  year,  spon¬ 
soring  military  services  forward  Candidate 
Nomination  Proposals  (CNPs)  for  systems 
to  be  evaluated  under  the  FCT  program  to 
the  DTE.  The  Services  are  encouraged  to 
prepare  and  submit  a  CNF  whenever  a 
promising  candidate  that  appears  to  satisfy 
a  current  or  potential  Service  requirement 
is  found.  A  CNP  must  contain  the  informa¬ 
tion  as  required  by  DOD  5000.3-M-2. 

The  fundamental  criterion  for  FCT  pro¬ 
gram  selection  is  the  candidate  system's 
potential  to  satisfy  operational  or  training 
requirements  that  exist  or  are  projected.  Its 
possible  contribution  to  the  U.S.  technol¬ 
ogy  base  is  considered  also.  Additional 
factors  influencing  candidate  selection  in¬ 
clude:  candidate  maturity,  available  test 
data,  multi-Service  interest,  existence  of  a 
statement  of  operational  requirement  need, 
potential  for  subsequent  procurement, 
sponsorship  by  U.S.-based  licensee,  realis¬ 
tic  evaluation  schedule  cost,  DOD  compo¬ 
nent  OSD  evaluation  cost-sharing  proposal, 
and  preprogrammed  procurement  funds. 
For  technology  evaluation  programs  within 
the  FCT  program,  the  candidate  nomina¬ 
tion  proposal  must  address  the  specific 
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arrangements  under  which  the  United 
States  and  foreign  participants  (govern¬ 
ments,  armed  forces,  corporations)  will 
operate.  These  may  include  govemment- 
to-govemment  Memoranda  of  Agreement, 
private  industry  licensing  agreements,  data 
exchange  agreements  and/or  cooperative 
technology  exchange  programs. 

Foreign  weapons  evaluation  activities  are 
funded  by  OSD  and  executed  by  the  Ser¬ 
vice  with  the  potential  need  for  the  system. 
Points  of  contact  at  the  headquarters  level 
in  each  Service  monitor  the  conduct  of  the 
programs.  Work  is  performed  in  laborato¬ 
ries  and  test  centers  throughout  the  coun¬ 
try.  Systems  evaluated  recently  under  the 
FCT  program  include  millimeter  wave  com¬ 
munications  equipment,  chemical  defense 
equipment,  gunnery  devices,  maritime  de¬ 
coys  and  navigational  systems. 

23.3  NATO  COMPARATIVE 
TEST  PROGRAM 

The  NATO  Comparative  Test  Program  has 
been  integrated  with  the  FCT  program.  It 
was  created  by  an  act  of  the  Congress  in  the 
FY  1986  Defense  Authorization  Bill.  The 
program  supports  the  evaluation  of  NATO 
nations'  weapons  systems,  equipment  and 
technology  and  assesses  their  suitability 
for  use  by  U.S.  forces.  The  selection  criteria 
for  the  NATO  Comparative  Test  Program 
are  essentially  the  same  as  for  the  FCT 
program.  The  exception  is  that  the  equip¬ 
ment  must  be  produced  by  a  NATO  mem¬ 
ber  nation  and  be  considered  as  an  alterna¬ 
tive  to  a  system  that  is  either  in  a  late  stage 
of  development  in  the  United  States  or  that 
offers  a  cost,  schedule  or  performance  ad¬ 
vantage  over  U.S.  equipment.  In  addition, 
the  NATO  Comparative  Test  Program  re¬ 
quires  that  notification  be  sent  to  the  Armed 
^rvices  and  Appropriations  Committees 
of  the  House  of  Representatives  and  Senate 
before  funds  are  obligated.  With  this  ex¬ 


ception,  the  NATO  Comparative  Test  Pro¬ 
gram  follows  the  same  nomination  process 
and  administrative  procedures.  Guidelines 
for  the  program  will  also  be  contained  in 
DOD  5000.3-M-2. 

Examples  of  proposals  funded  under  the 
NATO  Comparative  Test  Program  include 
T&E  of  a  German  mine  reconnaissance  and 
detection  system  for  the  Army,  a  United 
Kingdom-designed  minehunter  for  the 
Navy,  and  the  Norwegian  Penguin  missile 
system  for  the  Air  Force.  According  to  the 
FY  1988  Report  of  the  Secretary  of  Defense 
to  the  Congress,  the  program  has  gener¬ 
ated  considerable  interest  among  NATO 
allied  nations  and  has  become  a  primary 
way  of  promoting  armaments  cooperation 
within  NATO. 

Problems  associated  with  testing  foreign 
weapons  normally  stem  from  politics,  na¬ 
tional  pride  and  a  lack  of  previous  test  data. 
When  foreign  companies  introduce  weapon 
systems  for  testing,  they  often  will  attempt 
to  align  the  U.S.  military/  congressional  or¬ 
ganizations  with  their  systems.  For  ex¬ 
ample,  when  a  foreign  nation  introduced 
an  antitank  weapon  to  the  Army,  they  did 
so  by  having  a  U.S.  Senator  write  the  Army 
stating  a  need  for  the  system.  Attached  to 
the  letter  was  a  document  containing  doc¬ 
trine  to  employ  the  system  and  a  test  con¬ 
cept  to  use  when  evaluating  the  system. 
Systems  tested  in  the  NATO  Comparative 
Test  Program  often  become  involved  in 
national  pride.  The  test  community  must 
be  careful  not  to  allow  national  pride  to  be 
a  driving  force  in  the  evaluation.  At  times, 
the  9mm  pistol  competition  in  NATO  re¬ 
sembled  an  international  soccer  match,  with 
each  competing  nation  cheering  for  their 
pistol  and  many  other  nations  selecting 
sides.  Evaluating  the  9mm  pistol  was  dif¬ 
ficult  because  of  these  forces.  Thus,  U.S. 
testers  must  make  every  effort  to  obtain  all 
available  test  data  on  foreign  systems.  These 
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data  can  be  used  to  help  validate  the  evolv¬ 
ing  test  data  and  additional  test  data  dur¬ 
ing  the  evaluation. 

23.4  T&E  MANAGEMENT  IN 
MULTINATIONAL  PROGRAMS 

Rationalization,  standardization  and 
interoperability  have  become  increasingly 
important  elements  in  the  materiel  acquisi¬ 
tion  process.  Public  Law  94-361,  passed  on 
July  14, 1976,  requires  that  "equipment  for 
use  of  personnel  of  the  Armed  Forces  of  the 
United  States  stationed  in  Europe  under 
the  terms  of  the  North  Atlantic  Treaty 
should  be  standardized  or  at  least 
interoperable  with  equipment  of  other 
members  of  the  North  Atlantic  Treaty  Or¬ 
ganization"  (Reference  4,  pages  1-2).  Pro¬ 
gram  Managers  and  test  managers  must, 
therefore,  be  fully  aware  of  any  potential 
international  applications  of  the  systems 
for  which  they  are  responsible.  The  Joint 
Logistics  Commanders  Guide  for  the  Manage¬ 
ment  of  Multinational  Programs  published 
by  the  Defense  Systems  Management  Col¬ 
lege  (Reference  47)  is  a  valuable  compen¬ 
dium  of  information  for  the  program  man¬ 
ager  of  a  developing  system  with  potential 
multinational  applications. 

Representatives  of  the  United  States,  United 
Kingdom,  France  and  Germany  have  signed 
a  Memorandum  of  Agreement  concerning 
the  mutual  acceptability  of  each  country's 
T&E  data.  This  agreement  seeks  to  avoid 
redundant  testing  by  documenting  the  ex¬ 
tent  of  imderstanding  among  involved  gov¬ 
ernments  concerning  mutual  acceptability 
of  respective  T&E  procedures  for  systems 
that  are  developed  in  one  country  and  are 
candidates  for  procurement  by  one  or  more 
of  the  other  countries.  Focal  points  for  de¬ 
velopment  and  operational  testing  in  each 
of  the  countries  are  identified,  and  proce¬ 
dures  governing  generation  and  release  of 
T&E  data  are  described  in  the  Memoran¬ 
dum  of  Understanding  (MOU). 


Early  and  thorough  planning  is  an  impor¬ 
tant  element  of  any  successful  T&E  pro¬ 
gram  but  is  even  more  critical  in  a  multina¬ 
tional  program.  Agreement  must  be  reached 
concerning  T&E  procedures,  data  require¬ 
ments  and  methodology.  Differences  in 
tactics,  battlefield  representations  and  mili¬ 
tary  organizations  may  make  it  difficult  for 
one  nation  to  accept  another's  test  data. 
Therefore,  agreement  must  be  reached  in 
advance  concerning  the  operational  test 
scenario  and  battlefield  representation  that 
will  be  used. 

23.5  U.S.  AND  NATO  ACQUISITION 
PROGRAMS 

Some  test  programs  involve  combined  de¬ 
velopment  and  test  of  new  weapon  sys¬ 
tems  for  U.S.  and  other  NATO  countries.  In 
these  programs,  some  differences  from  the 
regular  "way  of  doing  things"  occur.  For 
example,  the  formulation  of  the  Request 
for  Proposal  (RFP)  must  be  coordinated 
with  the  North  Atlantic  Program  Manage¬ 
ment  Agency  (NAPMA);  and  their  input  to 
the  Statement  of  Work,  data  requirements, 
operational  test  planning  and  test  schedule 
formulation  must  be  included.  Also,  the 
U.S.  Army  operational  user.  Forces  Com¬ 
mand,  must  be  involved  in  the  operational 
test  program.  Usually,  a  Multinational 
Memorandum  of  Understanding  (MMOU) 
is  created  concerning  test  program  and  pro¬ 
duction  funding,  test  resources,  test  team 
composition,  use  of  national  assets  for  test¬ 
ing,  etc. 

Nations  are  encouraged  to  use  the  data  that 
another  nation  has  gathered  on  similar  test 
programs  to  avoid  duplication  of  effort. 
For  example,  during  the  U.S.  and  NATO 
Airborne  Warning  and  Control  System 
(AW ACS)  ESM  Program,  both  U.S.  and 
NATO  E-3  As  will  be  used  for  test  aircraft  in 
combined  development  test  and  evalua¬ 
tion  (DT&E)  and  subsequent  operational 
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test  and  evaluation  (OT&E).  Testing  will  be 
conducted  in  the  U.S.  and  European  the¬ 
aters.  The  Joint  Test  Force  will  be  com¬ 
posed  of  program  management  office,  con¬ 
tractor,  U.S.  operational  users.  Air  Force 
Operational  Test  and  Evaluation  Center 
(AFOTEC),  Force  Command  (NATO  us¬ 
ers),  and  logistics  personnel  for  this  pro¬ 
gram.  A  Multinational  Memorandum  of 
Agreement  for  this  program  was  created. 
The  U.S.  program  is  managed  by  the 
AW  ACS  System  Program  Office,  and  the 
NATO  program  is  managed  by  the 
NAPMA. 

23.6  SUMMARY 

The  procurement  of  weapon  systems  from 
foreign  nations  for  use  by  U.S.  Armed  Forces 


can  provide  the  following  advantages:  re¬ 
duced  research  and  development  costs, 
faster  initial  operational  capability,  im¬ 
proved  interoperability  with  friendly  na¬ 
tions,  and  lower  procurement  costs  because 
of  economies  of  scale.  Testing  such  systems 
presents  specific  challenges  to  accommo¬ 
date  the  needs  of  all  users.  Such  testing 
requires  careful  advance  planning  and  sys¬ 
tematic  execution.  Expectations  and  un¬ 
derstandings  must  be  well  documented  at 
an  early  stage  to  ensure  that  the  test  results 
have  utility  for  all  concerned. 
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NONDEVELOPMENT  ITEMS 


24.1  INTRODUCTION 

Many  options  are  available  when  an  acqui¬ 
sition  strategy  for  a  new  system  is  chosen. 
They  range  from  the  last  option  of  a  tradi¬ 
tional  new  research  and  development  pro¬ 
gram  to  modification  of  the  existing  sys¬ 
tem.  Between  these  two  extremes  are  other 
acquisition  strategies  that  call  for  using 
nondevelopment  items  to  various  extents. 
Figure  24-1,  an  adaptation  of  an  illustration 
found  in  Army  Materiel  Command  Pam¬ 
phlet  70-2,  shows  the  broad  spectrum  of 
approaches  that  can  be  taken  in  a  system 
acquisition  and  provides  examples  of  sys¬ 
tems  that  have  been  developed  using  each 
approach. 

24.1.1  Definition  of  NDI 

A  nondevelopmental  item  (NDI)  refers  to 
materiel  coming  from  a  variety  of  sources 
but  involving  little  or  no  development  ef¬ 
fort.  It  includes  commercial  products,  ma¬ 
teriel  developed  by  other  U.S.  government 
sources,  or  materiel  developed  in  other 
coimtries.  All  such  systems  are  required  to 
undergo  technical  and  operational  test  and 
evaluation  (T&E)  before  the  procurement 
decision,  unless  the  decision  authority 
makes  a  definitive  decision  that  previous 
testing  or  other  data  (such  as  user/market 
investigations)  provide  sufficient  evidence 
of  acceptability  (Reference  54).  A 
nondevelopmental  item  is: 


•  Any  previously  developed  item  in  use 
by  a  federal,  state  or  local  agency  of  the 
United  States  or  a  foreign  government 
with  which  the  United  States  has  a  mu¬ 
tual  defense  cooperation  agreement; 

•  Any  item  described  above  that  re¬ 
quires  only  minor  modification  to  meet 
the  requirements  of  the  procuring 
agency; 

•  Any  item  currently  being  produced 
that  does  not  meet  the  three  require¬ 
ments  above  solely  because  it  is  not  yet 
in  use  or  available  in  the  commercial 
marketplace.  (DODI  5000.2, 6-L) 

24.1.2  Advantages  and  Disadvantages 
of  the  NDI  Approach 

The  use  of  NDI  offers  the  following  advan¬ 
tages: 

•  The  time  to  field  a  system  is  greatly 
reduced,  and  a  quick  response  is  pro¬ 
vided  to  the  user's  needs; 

•  Research  and  development  costs  are 
reduced; 

•  State-of-the-art  technology  is  available 
immediately. 

NDI  offers  the  following  disadvantages: 

•  NDI  acquisitions  are  difficult  to  stan¬ 
dardize  with  the  current  fleet  equipment; 


•  Any  item  available  in  the  commercial 
marketplace; 
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MORE  COST 


Figure  24-1 .  The  Spectrum  of  Acquisition  Strategies 


•  NDI  acquisitio;.3  create  logistics  sup¬ 
port  difficulties; 

•  NDI  acquisitions  tend  not  to  have  com¬ 
petition;  therefore,  the  availability  of  sec¬ 
ond  source  is  not  present; 

•  With  NDI  acquisitions,  engineering  and 
test  data  often  is  not  available. 

24.1.3  Types  of  NDI 

Nondevelopment  items  can  be  separated 
into  two  general  categories;  each  requires  a 
modified  testing  approach.  The  categories 
are: 

(1)  Commercial  off-the-shelf  items  for 
use  in  the  same  environment  for  which 
the  items  were  designed .  Such  items  nor¬ 
mally  do  not  require  development  test¬ 
ing  prior  to  the  production  qualification 
test  except  in  those  cases  where  a  con¬ 
tract  may  be  awarded  to  a  contractor 
who  has  not  previously  produced  an 
acceptable  finished  product  and  the  item 
is  assessed  as  high  risk.  In  that  case, 
preproduction  qualification  testing 
would  be  required  (Reference  54). 

(2)  Commercial  off-the-shelf  items  for 
use  in  an  environment  other  than  that  for 
whirh  the  items  were  designed.  Such 
items  may  require  modifications  in  hard¬ 
ware  and/or  software.  These  items  re¬ 
quire  testing  in  an  operational  environ¬ 
ment,  preproduction  qualification  test 
ing  (if  previous  testing  resulted  in  item 
redesign),  and  production  qualification 
testing. 

Existing  components  that  must  be  inte¬ 
grated  with  a  new  system  configuration 
may  be  purchased  off  the  shelf.  These  would 
not  be  classified  as  NDIs,  but  many  of  the 
testing  and  evaluation  methods  would  still 
apply.  This  type  of  NDI  effort  requires 


more  extensive  research,  development  and 
testing  to  achieve  the  desired  system  con¬ 
figuration.  Testing  required  includes:  fea¬ 
sibility  testing  in  a  military  environment, 
preproduction  qualification  testing,  hard¬ 
ware/software  integration  testing,  opera¬ 
tional  testing  and  production  qualification 
testing. 

Given  the  variety  of  NDI  approaches  that 
may  be  employed,  it  is  imperative  that  the 
acquisition  strategy  clearly  specifies,  with 
the  agreement  of  the  testing  authority,  the 
level  of  testing  that  will  be  performed  on 
NDI  systems  and  the  environment  in  which 
those  systems  will  be  tested. 

24.2  MARKET  INVESTIGATION 
AND  PROCUREMENT 

A  market  investigation  is  the  central  activ¬ 
ity  leading  to  the  Milestone  I  review  deci¬ 
sion  regarding  the  use  of  an  NDI  acquisi¬ 
tion  strategy.  The  purpose  of  the  market 
investigation  is  to  determine  the  nature  of 
available  products  and  the  number  of  po¬ 
tential  vendors.  Market  investigations  may 
vary  from  informal  telephone  inquiries  to 
comprehensive  industry-wide  reviews. 
During  the  market  investigation,  sufficient 
data  must  be  gathered  to  support  a  defini¬ 
tive  NDI  decision,  to  finalize  the  require¬ 
ments  and  to  develop  an  acquisition  strat¬ 
egy  that  is  responsive  to  these  require¬ 
ments. 

During  the  Market  Investigation  Phase,  a 
formal  "request  for  information"  process 
may  be  followed  wherein  a  brief  narrative 
description  of  the  requirement  is  published 
and  interested  vendors  are  invited  to  re¬ 
spond.  Test  samples  or  test  items  may  be 
leased  or  purchased  at  this  time  to  support 
the  conduct  of  operational  suitability  tests, 
to  evaluate  the  ability  of  the  equipment  to 
satisfy  the  requirements  and  to  help  build 
the  functional  purchase  description  or  sys- 
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tern  specification.  This  type  of  preliminary 
testing  should  not  be  used  to  select  or  elimi¬ 
nate  any  part*:  -  ilar  vendor  or  product  un¬ 
less  it  is  prz  -  jded  by  competitive  contract¬ 
ing  pro*'  Jures  (Reference  61). 

It  is  imperative  that  technical  and  opera¬ 
tional  evaluators  become  involved  during 
this  early  stage  of  an  NDI  procurement  and 
that  they  perform  an  early  assessment  of 
the  initial  issues.  The  evaluator  must  also 
relate  these  issues  to  test  and  evaluation 
criteria  and  provide  their  independent 
evaluation  plans  and  reports  to  the  deci¬ 
sion  authorities  before  the  Milestone  I  deci¬ 
sion  review. 

24.3  NDI  TESTING 

24.3.1  General  Considerations 

Test  and  evaluation  must  be  considered 
throughout  the  acquisition  of  a  system  that 
involves  NDI.  The  program  manager  and 
his  staff  must  ensure  that  the  testing  com¬ 
munity  is  fully  involved  in  the  acquisition 
from  the  start.  The  amount  and  level  of 
testing  required  depends  on  the  nature  of 
the  NDI  and  its  anticipated  use;  it  should  be 
planned  to  support  the  design  and  decision 
process.  At  a  minimum,  T&E  of  NDI  will  be 
conducted  to  verify  integration  and 
interoperability  with  other  system  elements. 
All  NDI  modifications  necessary  to  adapt 
them  to  the  weapon  system  environment 
will  also  be  subject  to  T&E.  Available  test 
results  from  all  commercial  and  govern¬ 
ment  sources  will  determine  the  actual  ex¬ 
tent  of  testing  necessary.  There  are  some 
inherent  advantages  in  NDI  testing.  For 
example,  an  NDI  usually  encompasses  a 
mature  design.  The  availability  of  this  ma¬ 
ture  design  contributes  to  the  rapid  devel¬ 
opment  of  the  logistics  support  system  that 
will  be  needed.  In  addition,  there  are  more 
"production"  items  available  for  use  in  a 
test  program.  The  program  manager  and 


his  staff  must  remember  that  NDI  systems 
also  require  activity  in  areas  associated  with 
traditional  development  and  acquisition 
programs.  For  example,  training  and  main¬ 
tenance  programs  and  manuals  must  be 
developed;  and  sufficient  time  should  be 
allowed  for  their  preparation. 

When  the  solicitation  package  for  an  NDI 
acquisition  is  assembled,  the  program  man¬ 
ager  must  ensure  that  it  includes  the  fol¬ 
lowing  T&E-related  items: 

(1)  Approved  T&E  issues  and  criteria; 

(2)  A  requirement  that  the  offerer  pro¬ 
vide  a  description  of  the  testing  per¬ 
formed  by  the  contractor  on  the  system, 
including  test  procedures  followed,  data 
and  results  achieved; 

(3)  Production  qualification  test  and  qual¬ 
ity  conformance  requirements; 

(4)  Acceptance  test  plans  for  the  system 
and  its  components. 

24.3.2  Testing  Before  Milestone  I 

An  important  advantage  of  using  an  NDI 
acquisition  strategy  is  reduced  acquisition 
time.  Consequently,  it  is  important  that 
testing  not  be  redundant  and  that  it  is  lim¬ 
ited  to  the  minimum  effort  necessary  to 
obtain  the  required  data.  Testing  can  be 
minimized  by: 

(1)  Obtaining  and  assessing  contractor 
test  results; 

(2)  Obtaining  usage/failure  data  from 
other  customers; 

(3)  Observing  contractor  testing; 

(4)  Obtaining  test  results  from  indepen¬ 
dent  test  organizations  (e.g..  Under¬ 
writer's  Laboratory); 
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(5)  Verifying  selected  contractor  test  data. 

If  it  is  determined  that  more  information  is 
needed  after  the  initial  data  collection  from 
the  above  sources,  NDI  candidates  may  be 
bought  or  leased,  and  technical  and  opera¬ 
tional  tests  may  be  conducted. 

24.3.3  Testing  After  Milestone  I 

All  testing  to  be  conducted  after  the  initial 
milestone  decision  to  proceed  with  the  NDI 
acquisition  should  be  described  in  the  Ac¬ 
quisition  Strategy  and  the  Test  and  Evalu¬ 
ation  Master  Plan.  Development  testing  is 
conducted  only  if  specific  information  that 
cannot  be  satisfied  by  contractor  or  other 
test  data  sources  is  needed.  Operational 
testing  is  conducted  as  needed.  The  inde¬ 
pendent  operational  test  and  evaluation 
agency  should  concur  in  any  decisions  to 
limit  or  eliminate  operational  testing. 

Test  and  evaluation  continue  even  after  the 
system  has  been  fielded.  This  testing  takes 
the  form  of  a  follow-on  evaluation  to  vali¬ 
date  and  refine:  operating  and  support  cost 
data;  reliability,  availability,  and  maintain¬ 
ability  characteristics;  logistic  support 
plans;  and  training  requirements,  doctrine 
and  tactics. 

24.4  RESOURCES  AND  FUNDING 

Programming  and  budgeting  for  an  NDI 
acquisition  present  a  special  challenge.  Be¬ 
cause  of  the  short  duration  of  the  NDI 
acquisition  process,  the  standard  lead  times 
required  in  the  normal  Planning,  Program¬ 
ming,  and  Budgetary  System  cycle  may  be 
unduly  restrictive.  This  situation  can  be 
minimized  through  careful,  advanced  plan¬ 
ning  and,  in  the  case  of  urgent  require¬ 


ments,  reprogramming/supplemental 
funding  techniques. 

Research,  Development,  Test  and  Evalua¬ 
tion  (RDT&E)  funds  are  normally  used  to 
support  the  conduct  of  the  Market  Investi¬ 
gation  Phase  and  the  purchase  or  lease  of 
NDI  candidates  required  for  T&E  purposes. 
The  RDT&E  funds  are  also  used  to  support 
T&E  activities  such  as:  modification  of  the 
NDI;  purchase  of  specifications,  man¬ 
ufacturer's  publications,  repair  parts,  spe¬ 
cial  tools  and  equipment;  transportation  of 
the  NDI  to  and  from  the  test  site;  and  train¬ 
ing,  salaries  and  temporary  duty  costs  of 
T&E  personnel.  Procurement,  operations 
and  maintenance  funds  are  usually  used  to 
support  production  and  deployment  costs. 

One  chief  advantage  of  using  an  NDI  ac¬ 
quisition  strategy  is  reduced  overall  cost. 
Additional  cost  savings  can  be  achieved 
after  a  contract  has  been  awarded  if  the 
program  manager  ensures  that  incentives 
are  provided  to  contractors  to  submit  value 
engineering  change  proposals  to  the  gov¬ 
ernment  when  unnecessary  costs  are  iden¬ 
tified. 

24.5  SUMMARY 

The  use  of  nondevelopment  items  in  a  sys¬ 
tem  acquisition  can  provide  considerable 
time  and  cost  savings.  The  testing  approach 
used  for  an  NDI  acquisition  must  be  care¬ 
fully  tailored  to  the  type  of  system  and  the 
amount  of  test  data  already  available.  The 
T&E  community  must  get  involved  early  in 
the  process  so  that  all  test  issues  are  ad¬ 
equately  addressed  and  timely  compre¬ 
hensive  evaluations  are  provided  to  deci¬ 
sion  authorities. 
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25 

TESTING  THE  SPECIAL  CASES 


25.1  INTRODUCTION 

This  chapter  covers  the  special  factors  and 
alternative  test  strategies  the  tester  must 
consider  in  testing  dangerous  or  lethal 
weapons,  systems  that  involve  one-of-a- 
kind  or  limited  production  and  systems 
with  high-cost  and  /  or  special  security  con¬ 
siderations.  Exeimples  include  chemical  and 
laser  weapons;  ships;  space  weapons;  mis¬ 
sile  systems;  and  electronic  warfare  (EW), 
conunand  and  control  (C2)  and  intelligence 
systems. 

25.2  TESTING  OF  HAZARDOUS 
WEAPONS 

The  tester  of  dangerous  or  lethal  systems, 
like  chemical  and  laser  weapons,  must  con¬ 
sider  various  safety,  health  and  medical 
factors  in  developing  test  plans,  such  as: 

(1)  Provision  of  medical  facilities  for  pre- 
and  post-test  checkups  and  emergency 
treatment; 

(2)  Need  for  protective  gear  for  partici¬ 
pating/observer  personnel; 

(3)  Approval  of  the  test  plan  by  the  Sur¬ 
geon  General; 

(4)  Restrictions  in  selection  of  test  par¬ 
ticipants  (e.g.,  medical  criteria  or  use  of 
only  volimteer  troops); 

(5)  Restricted  test  locations; 


(6)  Environmental  Impact  Statements. 

Also,  the  tester  must  allow  for  additional 
planning  time,  test  funds  and  test  resources 
to  accommodate  such  factors. 

25.2.1  Chemical  Weapons  Testing 

The  testing  of  chemical  weapons  poses 
unique  problems,  because  the  tester  cannot 
perform  actual  open-air  field  testing  with 
real  nerve  agents  or  other  toxic  chemicals. 
Since  the  United  States  signed  and  ratified 
the  Geneva  Protocol  of 1925,  U.S.  policy  has 
been  that  the  United  States  will  never  be  the 
first  to  use  lethal  chemical  weapons;  it  may, 
however,  retaliate  with  chemical  weapons 
if  so  attacked.  In  addition  to  the  health  and 
safety  factors  discussed  in  the  last  para¬ 
graph,  test  issues  the  chemical  weapons 
tester  must  address  include: 

(1)  All  possible  chemical  reactions  due  to 
variations  such  as  moisture,  tempera¬ 
ture,  pressure  and  contamination; 

(2)  Physical  behavior  of  the  chemical;  i.e., 
droplet  size,  dispersion  density  and 
ground  contamination  pattern  when 
used  operationally; 

(3)  Toxicity  of  the  chemical;  i.e.,  lethality 
and  duration  of  contamination  when 
used  operationally; 


25-1 


(4)  Safety  of  the  chemical  weapon  during 

storage,  handling  and  delivery; 

(5)  Decontamination  process. 

Addressing  all  of  these  issues  requires  a 
combination  of  laboratory  toxic  chamber 
tests  and  open-air  field  testing.  The  latter 
must  be  performed  using  "simulants," 
which  are  substances  that  replicate  the 
physical  and  chemical  properties  of  the 
agent  but  with  no  toxicity. 

The  development  and  use  of  simulants  for 
testing  will  require  increased  attention  as 
more  chemical  weapons  are  developed. 
Chemical  agents  can  demonstrate  a  wide 
variety  of  effects  depending  on  such  factors 
as  luoisture,  temperature  and  contamina¬ 
tion.  Consequently,  the  simulants  must  be 
able  to  replicate  all  possible  agent  reac¬ 
tions;  it  is  likely  that  several  simulants 
would  have  to  be  used  in  a  test  to  produce 
all  predicted  agent  behaviors.  In  develop¬ 
ing  and  selecting  simulants,  the  tester  must 
thoroughly  understand  all  chemical  and 
physical  properties  and  possible  reactions 
of  the  agent. 

Studies  of  the  anticipated  reactions  can  be 
performed  in  toxic-chamber  tests  using  the 
real  agent.  Here,  factors  such  as  changes  in 
moisture,  temperature,  pressure  and  levels 
of  impurity  can  be  controlled  to  assess  the 
agent's  behavior.  But,  the  tester  must  think 
through  all  possible  environmental  condi¬ 
tions  in  which  the  weapon  could  operate  so 
all  cases  can  be  tested  in  the  laboratory 
chamber  with  the  real  agent.  For  example, 
during  development  testing  of  the  BIGEYE 
chemical  weapon,  it  was  found  thathigher- 
than-expected  temperatures  due  to  aero¬ 
dynamic  heating  caused  pressure  buildup 
in  the  bomb  body  that  resulted  in  the  bomb 
exploding.  This  caused  the  operational  con¬ 
cept  for  the  BIGEYE  to  be  changed  from  on¬ 
board  mixing  of  the  two  chemicals  to  mix¬ 
ing  after  release  of  ;he  bomb. 


Tests  to  confirm  toxicity  must  be  conducted 
using  simulants  in  the  actual  environment. 
Since  the  agent's  toxicity  is  dependent  on 
factors  such  as  droplet  size,  dispersion  den¬ 
sity,  ground  contamination  pattern  and 
degradation  rate,  a  simulant  that  behaves 
as  the  agent  does  must  be  used  in  actual 
field  testing.  Agent  toxicity  is  determined 
in  the  lab. 

The  Services  publish  a  variety  of  technical 
documents  on  specific  chemical  test  proce¬ 
dures.  Documents  such  as  the  U.S.  Army 
Test  and  Evaluation  Command  (TECOM) 
Pamphlet  310-4,  a  bibliography  that  in¬ 
cludes  numerous  reports  on  chemical  test¬ 
ing  issues  and  procedures,  can  be  consulted 
for  specific  documentation  on  chemical  test¬ 
ing. 

25.2.2  Laser  Weapons  Testing 

Many  new  weapon  systems  are  being  de¬ 
signed  with  embedded  laser  range  finders 
and  laser  designators.  Because  of  the  dan¬ 
ger  to  the  human  eye  posed  by  lasers,  the 
tester  must  adhere  to  special  safety  require¬ 
ments  and  utilize  special  locations  during 
test  and  evaluation  (T&E).  For  instance,  the 
only  Army  installation  in  the  continental 
United  States  permitting  free-play  airborne 
laser  testing  is  Fort  Hunter-Liggett,  Calif. 
During  tests  involving  lasers,  the  airspace 
must  be  restricted;  and  guards  must  be 
posted  to  prevent  anyone  from  acciden¬ 
tally  venturing  into  the  area.  A  potential 
solution  to  the  safety  issue  is  to  develop 
and  use  an  "eye-safe"  laser  for  testing.  The 
tester  must  ensure  that  eye-safe  lasers  pro¬ 
duce  the  same  laser  energy  as  the  real  laser 
system. 

Another  concern  of  the  laser  energy  weap¬ 
ons  tester  is  the  accurate  determination  of 
laser  energy  level  and  location  on  the  tar¬ 
get.  Measurements  of  the  laser  energy  on 
the  target  are  usually  conducted  in  the 
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laboratory  as  part  of  development  test  (DT) . 
In  the  field,  video  cameras  are  often  used  to 
verify  that  the  laser  designator  did  indeed 
illuminate  the  target.  Such  determinations 
are  important  when  the  tester  is  trying  to 
attribute  weapon  performance  to  behavior 
of  the  laser,  behavior  of  the  guidance  sys¬ 
tem,  or  some  other  factor. 

A  bibliography  of  Army  test  procedures, 
TECOM  Pamphlet  3 10-4,  lists  several  docu¬ 
ments  that  cover  the  special  issues  associ¬ 
ated  with  laser  testing. 

25.3  SPACE-SYSTEM  TESTING 

From  a  historical  perspective,  space-sys¬ 
tem  acquisition  has  posed  several  unique 
problems  to  the  test  process  (especially  the 
operational  test  process)  that  generally  fall 
into  four  categories:  limited  quantities/high 
cost,  "block  upgrade"  approach  to  acquisi¬ 
tion,  operating  environment  (peacetime  and 
wartime),  and  test  environment. 

(1)  Limited  quantities /high  cost  -  Space 
systems  have  traditionally  involved  the 
acquisition  of  relatively  few  (historically, 
less  than  20)  systems  at  extremely  "high 
per-unit  costs"  (in  comparison  with  more 
traditional  military  systems).  The  high 
per-unit  costs  are  driven  by  a  combina¬ 
tion  of  high  transportation  costs  (launch 
to  orbit),  high  life-cycle  reliability  require¬ 
ments  and  associated  costs  because  of 
the  lack  of  an  "on-orbit"  maintenance 
capability  and  the  high  costs  associated 
with  "leading  edge"  technologies  that 
tend  to  be  a  part  of  spacecraft  design. 
From  a  test  perspective,  this  serves  to 
drive  space-system  acquisition  strategy 
into  the  "nonstandard"  approach  ad¬ 
dressed  below.  The  problem  is  com¬ 
pounded  by  the  "block  upgrade"  ap¬ 
proach  to  acquisition. 

(2)  Block  upgrade  approach  to  acquisi¬ 
tion-  Due  to  the  "limited  buy"  and  "high 


per-unit  cost"  nature  of  spacecraft  acqui¬ 
sition,  these  systems  tend  to  be  procured 
using  a  "block  upgrade"  acquisition  strat¬ 
egy.  Under  this  concept,  "the  decision  to 
deploy"  is  often  made  at  the  front  end  of 
the  acquisition  cycle;  and  the  first  proto¬ 
type  to  be  placed  in  orbit  becomes  the 
first  operational  asset.  As  early  and  fol¬ 
low-on  systems  undergo  ground  and  on- 
orbit  testing  (either  development  test  and 
evaluation  (DT&E)  or  operatonal  test  and 
evaluation  (OT&E)),  discrepancies  are 
corrected  by  "block  changes"  to  the  next 
system  in  the  pipeline.  This  approach  to 
acquisition  can  perturb  the  test  process 
as  the  tester  may  have  no  formal  mile¬ 
stone  decisions  to  test  toward.  The  focus 
must  change  toward  being  able  to  influ¬ 
ence  the  design  of  (and  block  changes  to) 
systems  further  downstream  in  the  pipe¬ 
line.  As  the  first  "on-orbit"  asset  usually 
becomes  the  first  operational  asset,  pres¬ 
sure  is  created  from  the  operational  com¬ 
munity  to  expedite  (and  sometimes  limit) 
testing  so  a  limited  operational  capabil¬ 
ity  can  be  declared  and  the  system  can 
begin  fulfilling  mission  requirements. 
Once  the  asset  "goes  operational,"  any 
use  of  it  for  testing  must  compete  with 
operational  mission  needs  —  a  situation 
potentially  placing  the  tester  in  a  posi¬ 
tion  of  relatively  low  priority.  Recogni¬ 
tion  of  these  realities  and  careful  "early- 
on"  test  planning  can  overcome  many  of 
these  problems,  but  the  tester  needs  to  be 
involved  and  ready  much  earlier  in  the 
cycle  than  with  traditional  systems. 

(3)  Operating  environment  fpeacetime 
and  wartime)  -  Most  currently  deployed 
space  systems  and  near-term  future  space 
systems  operate  in  the  military  support 
arena  such  as  tactical  warning /attack 
assessment,  communications,  naviga¬ 
tion,  weather  and  intelligence;  and  their 
day-to-day  peacetime  operating  environ¬ 
ment  is  not  much  different  from  the  war- 
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time  operating  environment  except  for 
activity  level  (i.e.,  message  throughput, 
more  objects  to  track/see,  etc.)-  Histori¬ 
cally,  space  has  been  a  relatively  benign 
battlefield  environment  because  of  tech¬ 
nology  limitations  in  the  capability  of 
potential  adversaries  to  reach  into  space 
with  weapons.  This  combination  of  sup- 
port-type  missions  and  a  battlefield  en¬ 
vironment  that  is  not  much  different  from 
the  peacetime  environment  has  played  a 
definite  role  in  allowing  systems  to  reach 
limited  operational  capability  without 
as  much  dedicated  prototype  system- 
level  testing  as  seen  on  other  systems. 
However,  this  situation  is  likely  to  change 
with  the  ad\  *  of  systems  like  the  Stra¬ 
tegic  Defense  Initiative  (SDI)  where  ac¬ 
tual  weapons  systems  are  on  alert  in 
space,  and  day-to-day  peacetime  opera¬ 
tions  will  not  mirror  the  anticipated 
battlefield  environment  as  closely.  Like¬ 
wise,  the  elevation  of  the  battlefield  into 
space  and  the  advancing  technologies 
that  allow  potential  adversaries  to  reach 
into  space  may  also  change  the  thrust  of 
how  space  systems  need  to  be  tested  in 
space.  An  increased  need  for  dedicated 
on-orbit  testing  on  a  type  of  space  range 
where  the  battlefield  environment  will 
be  replicated  can  be  anticipated — a  situ¬ 
ation  similar  to  the  dedicated  testing  done 
today  on  test  ranges  with  Army,  Navy 
and  Air  Force  weapons. 

(4)  Test  environment  -  The  location  of 
space  assets  in  “remote"  orbits  also  com¬ 
pounds  the  test  problem.  Space  systems 
do  not  have  the  ready  access  (as  with 
ground  or  aircraft  systems)  to  correct 
deficiencies  identified  during  testing. 
This  situation  has  driven  the  main  thrust 
of  testing  into  the  "prelaunch"  ground 
simulation  environment  where  discrep¬ 
ancies  can  be  corrected  before  the  system 
becomes  inaccessible.  However,  as  men¬ 
tioned  previously,  when  space-system 


missions  change  from  a  war-support  fo¬ 
cus  to  a  war-fighting  focus  and  the  num¬ 
ber  of  systems  required  to  do  the  mission 
increases  from  the  "high  reliability /lim¬ 
ited  number"  mode  to  a  more  traditional 
"fairly  large  number  buy"  mode,  future 
space-system  testing  could  be  expected 
to  become  more  like  the  testing  associ¬ 
ated  with  current  ground,  sea  and  air 
systems.  From  a  test  perspective,  this 
could  also  create  unique  "test  technol¬ 
ogy"  requirements;  i.e.,  with  these  sys¬ 
tems  we  will  have  to  bring  the  test  range 
to  the  operating  system  as  opposed  to 
bringing  the  system  to  the  range.  Also, 
because  the  space  environment  tends  to 
be  "visible  to  the  world"  (others  can  ob¬ 
serve  our  tests  as  readily  as  we  can), 
unique  test  operations  security  method¬ 
ologies  may  be  required  to  allow  us  to 
achieve  test  realism  without  giving  away 
system  vulnerabilities. 

In  summary,  current  and  near-term  future 
space  systems  have  unique  test  method¬ 
ologies.  However,  in  the  future,  space  op¬ 
erations  might  entail  development /deploy¬ 
ment  of  weapon  platforms  on  orbit  with 
lower  design-life  reliability  (because  of 
cost);  and  day-to-day  peacetime  operations 
will  not  mirror  the  wartime  environment. 
Thus,  space-system  testing  requirements 
may  begin  to  more  closely  parallel  those  of 
traditional  weapon  systems. 

25.4  TESTING  WITH  LIMITATIONS 

Certain  types  of  systems  cannot  be  tested 
using  relatively  standard  T&E  approaches 
for  reasons  such  as  a  nonstandard  acquisi¬ 
tion  strategy,  resource  limitations,  cost, 
safety  or  security  constraints.  The  Test  and 
Evaluation  Master  Plan  (TEMP)  must  con¬ 
tain  a  statement  that  identifies  "those  fac¬ 
tors  that  will  preclude  a  full  and  completely 
realistic  operational  test...(IOT&E  and 
FOT&E),"  such  as  inability  to  realistically 
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portray  the  entire  threat,  limited  resources 
or  locations,  safety  and  system  maturity. 
The  impact  of  these  limitations  on  the  test's 
critical  operational  issues  must  also  be  ad¬ 
dressed  in  the  TEMP. 

Nonstandard  acquisition  strategies  are  of¬ 
ten  used  for  one-of-a-kind  or  limited  pro¬ 
duction  systems.  Examples  of  these  include 
space  systems;  missiles;  ships;  and  EW,  C2 
and  intelligence  systems.  For  one-of-a-kind 
systems,  the  production  decision  is  often 
made  prior  to  system  design;  hence,  testing 
does  not  support  the  traditional  decision 
process.  In  limited  production  systems, 
there  are  often  no  prototypes  available  for 
test;  consequently,  the  tester  must  develop 
innovative  test  strategies. 

25.5  OPERATIONS  SECURITY 
AND  T&E 

Operations  security  (OPSEC)  issues  must 
be  considered  in  all  test  planning.  The  DODI 
5000.2  requires  the  protection  of  "sensitive 
design  information  and  test  data"  through¬ 
out  the  acquisition  cycle  by: 

(1)  Protecting  sensitive  technology; 

(2)  Eliminating  nonsecure  transmittal 
data  on  and  from  test  ranges; 

(3)  Providing  secure  communications 
linking  DOD  agencies  to  each  other  and 
to  their  contractors. 

Such  protection  is  obviously  costly  and 
will  require  additional  planning  time,  test 


resources  and  test  constraints.  The  test  plan¬ 
ner  must  determine  all  possible  ways  in 
which  the  system  could  be  susceptible  to 
hostile  exploitation  during  testing.  For  ex¬ 
ample,  announcement  of  test  schedule  and 
location  could  allow  monitoring  by  unau¬ 
thorized  persons.  Knowledge  of  the  loca¬ 
tions  of  systems  and  instrumentation  or 
test  conce;  ts  could  reveal  classified  system 
capabilities  or  military  concepts.  Compila¬ 
tions  of  unclassified  data  could,  as  a  whole, 
reveal  classified  information  as  could  sur¬ 
veillance  (electronic  or  photographic)  of 
test  activities  or  interception  of  unencrypted 
transmissions.  The  T &E  regulations  of  each 
Service  require  an  operational  security  plan 
for  a  test.  A  detailed  list  of  questions  the  test 
planner  can  use  to  identify  the  potential 
threat  of  exploitation  is  provided  in  AFR 
55-43. 

25.6  SUMMARY 

All  weapon  systems  tests  are  limited  to 
some  degree,  but  certain  systems  face  ma¬ 
jor  limitations  that  could  preclude  a  full 
and  realistic  test.  The  test  planners  of  these 
special  systems  must  allow  additional  plan¬ 
ning  time,  budget  for  extra  test  resources 
and  devise  alternative  test  strategies  to  work 
around  testing  limitations  caused  by  such 
factors  as  security  restrictions,  resource 
availability,  environmental  safety  factors 
and  nonstandard  acquisition  strategies. 
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T&E  OF  WEAPON  SYSTEMS  TYPES 


26.1  INTRODUCTION 

This  chapter  offers  guidance  to  Depart¬ 
ment  of  Defense  personnel  who  plan,  moni¬ 
tor  and  execute  test  and  evaluation  (T&E). 
Checklists  for  the  chapter  were  obtained 
from  the  Defense  Science  Board  Study,  Re¬ 
port  of  Task  Force  on  Test  and  Evaluation, 
dated  April  2, 1974.  This  excellent  study  is 
highly  regarded  in  the  T &E  community  but 
has  become  dated;  consequently,  the  De¬ 
fense  Systems  Management  College  de¬ 
cided  to  update  the  study  findings  and 
include  those  findings  and  summary  check¬ 
lists  in  this  management  guide. 

26.2  GENERAL  TEST 
AND  EVALUATION  ISSUES 

The  Defense  Science  Board  (DSB)  report 
presented  guidance  on  T&E  at  two  levels. 
On  a  general  level  it  discussed  a  number  of 
issues  that  were  appropriate  to  all  weapon 
acquisition  programs.  These  issues,  along 
with  a  summary  discuss v  n  are  given  be¬ 
low. 

26.2.1  Effects  of  Test  Requirements  on 
System  Acquisition 

The  acquisition  strategy  for  the  system 
should  allow  sufficient  time  between  the 
end  of  demonstration  testing  and  procure¬ 
ment,  as  contracted  with  limited  produc¬ 
tion  decisions,  to  allow  flexibility  for  modi¬ 
fication  of  plans  that  will  be  reqi  v 'd. 
should  ensure  that  sufficient  dollars  are 
available  not  only  to  conduct  T&E  but  to 
allow  for  additional  T&E  that  is  always 


required  due  to  failure,  design  changes, 
etc.;  and,  it  should  be  evaluated  relative  to 
constraints  imposed  by: 

•  The  level  of  system  testing  at  various 
stages  of  the  research,  development,  test 
and  evaluation  (RDT&E)  cycle; 

•  The  number  of  test  items  available  and 
the  schedule  interface  with  other  sys¬ 
tems  needed  in  the  tests,  such  as  aircraft, 
electronics,  etc.; 

•  The  support  required  to  assist  in  pre¬ 
paring  for  and  conducting  tests  and  ana¬ 
lyzing  the  test  results; 

•  Being  evaluated  to  minimize  the  so- 
called  T&E  gap  caused  by  lack  of  hard¬ 
ware  during  the  test  phase. 

26.2.2  T est  Requirements  and  Restrictions 

Tests  should: 

•  Have  specific  objectives; 

•  List,  in  advance,  actions  to  be  taken  as 
a  consequence  of  the  test  results; 

•  Be  instrumented  to  permit  diagnosis  of 
tht^  cause  cf  lack  of  performance  includ¬ 
ing  random,  design  induced,  wear  out 
and  operator  error  failure; 

•  If  failures  occur,  not  be  repeated  with¬ 
out  a  detailed  analysis  of  the  failure. 


("Most  likely  the  failure  will  not  go 
away.") 

26.2.3  Trouble  Indicators 

Establish  an  early  detection  scheme  to  iden¬ 
tify  program  illness. 

When  a  program  begins  to  have  trouble, 
there  are  indicators  that  will  show  up  dur¬ 
ing  testing.  Some  of  these  indicators  are: 

•  A  test  failure; 

•  Any  repetitive  failure; 

•  A  revision  of  schedule  or  incremental 
funding  that  exceeds  the  original  plan; 

•  Any  relaxation  of  the  basic  require¬ 
ments  such  as  lower  performance. 

26.2.4  Requirement  For  Test  Rehearsals 

Test  rehearsals  should  be  conducted  for 
each  new  phase  of  testing. 

26.3  SCHEDULING 

Specific  issues  associated  with  test  sched¬ 
uling  are  listed  below. 

26.3.1  Building  Block  Test  Scheduling 

The  design  of  a  set  of  tests  to  demonstrate 
feasibility  prior  to  the  Engineering  and 
Manufacturing  Development  Phase  should 
be  used.  This  will  allow  early  testing  of 
high-technical-risk  items,  and  subsequent 
tests  can  be  incorporated  into  the  hardware 
as  the  system  concept  has  been  demon¬ 
strated  as  feasible. 

26.3.2  Component  and  Subsystem  Test 
Plans 

Ensure  a  viable  component  and  subsystem 
test  plan.  Studies  show  that  almost  all  com¬ 


ponent  failures  will  be  the  kind  that  cannot 
be  easily  detected  or  prevented  in  full  sys¬ 
tem  testing.  System  failure  must  be  de¬ 
tected  and  fixed  in  the  component /sub¬ 
system  stage,  as  detecting  and  correcting 
failure  only  at  the  operational  test  level 
results  in  high  cost. 

26.3.3  Phasing  of  DT&E  and  lOT&E 

Problems  that  become  apparent  in  opera¬ 
tional  testing  can  often  be  evaluated  faster 
with  the  instrumented  development  test 
and  evaluation  (DT&E)  hardware.  The  in¬ 
tegrated  test  plan  should  provide  time  and 
money  to  investigate  test  failures  and  elimi¬ 
nate  causes  of  failures  before  other,  similar 
tests  take  place. 

26.3.4  Schedule  lOT&E  to  Include 
System  Interfaces  with  Other  Systems 

Whenever  possible,  the  initial  operational 
test  and  evaluation /follow-on  operational 
test  and  evaluation  (IOT&E)/(FOT&E)  of  a 
weapon  system  should  be  planned  to  in¬ 
clude  other  systems  that  must  have  a  tech¬ 
nical  interface  with  the  new  system.  For 
example,  the  missile  should  be  tested  on 
most  of  the  platforms  for  which  they  are 
programmed. 

26.4  RESOURCES  FOR  TESTING 

26.4.1  Identify  Test  Resources  and 
Instrumentation 

As  early  as  possible,  but  not  later  than  the 
start  of  the  Engineering  and  Manufactur¬ 
ing  Development  Phase,  the  test  facilities 
and  instrumentation  requirements  to  con¬ 
duct  operational  tests  should  be  identified 
and  a  tentative  schedule  of  test  activities 
prepared.  This  information  is  recorded  in 
the  Test  and  Evaluation  Master  Plan  (TEMP) 
and  Service  test  resource  documentation. 
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26.4.2  Require  Multi-Service  OT&E 

Multi-Service  operational  test  and  evalua¬ 
tion  (OT&E)  should  be  considered  for 
weapon  systems  requiring  new  operational 
concepts  involving  other  Services.  If  multi- 
Service  testing  is  used,  an  analysis  of  the 
impact  of  demonstration  on  time  and  re¬ 
sources  needed  to  execute  the  multi-Ser- 
vice  tests  should  be  conducted  before  the 
Milestone  II  decision. 

26.4.3  Military  Construction  Program 
Facilities 

Some  programs  cannot  be  tested  without 
Military  Construction  Program  facilities. 
To  construct  these  facilities  will  require 
long  lead  times;  therefore,  early  planning 
must  be  done  to  ensure  that  the  facilities 
will  be  ready  when  required. 

26.4.4  Test  Sample  Size 

The  primary  basis  for  the  test-sample  size  is 
usually  based  on  one  or  more  of  the  follow¬ 
ing: 

•  Analysis  of  test  objectives; 

•  Statistical  significance  of  test  results  at 
some  specified  confidence  level; 

•  Availability  of  test  vehicles,  items,  etc.; 

•  Support  resources  or  facilities  avail¬ 
able; 

•  Time  available  for  the  test  program. 

26.4.5  Test  Termination 

One  should  not  hesitate  to  terminate  a  test 
before  its  completion  if  it  becomes  clear 
that  the  main  objective  of  the  test  is 
unachievable  (due  to  hardware  failure,  un¬ 
availability  of  resources,  etc.)  or  if  addi¬ 


tional  samples  will  not  change  the  outcome 
and  conclusions  of  the  test. 

26.5  COST 

26.5.1  Budget  for  Test 

The  IPS,  TEMP  and  budgeting  documents 
should  be  reviewed  regularly  to  ensure 
that  there  are  adequate  identified  testing 
funds  relative  to  development  and  fabrica¬ 
tion  funds. 

26.5.2  Funds  for  Correcting  Faults 
Found  in  Testing 

The  IPS,  TEMP  and  budgeting  documents 
need  careful  scrutiny  to  ensure  that  there 
are  adequate  contingency  funds  to  cover 
correction  of  difficulties  at  a  level  that 
matches  industry  /  government  experience 
on  the  contract.  (Testing  to  correct  deficien¬ 
cies  found  during  testing,  without  suffi¬ 
cient  funding  for  proper  correction,  results 
in  Band-Aid  approaches,  which  require 
corrections  at  a  later  and  more-expensive 
time  period.) 

26.6  PERFORMANCE  AND 
OPERATIONAL  ISSUES 

26.6.1  Proof  of  Performance  of 
Human  Factors  Concepts 

At  an  appropriate  time  in  Concept  Explora¬ 
tion  and  Definition  or  Demonstration  and 
Validation  (DEM/VAL)  Phases,  T&E 
should  authenticate  the  human  factors  con¬ 
cepts  embodied  in  the  proposed  systems 
design,  examining  questions  of  safety,  com¬ 
fort,  man-machine  interfaces,  as  well  as  the 
number  and  skill  of  personnel  required. 

26.6.2  Test  Planning 

A  summary  of  important  test  planning 
items  that  were  identified  by  the  DSB  is 
provided  below: 
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•  Ensure  that  the  whole  system,  includ¬ 
ing  the  system  user  personnel,  is  tested. 
Realistically  test  the  complete  system, 
including  hardware,  software,  people 
and  all  interfaces.  Get  user  involved  from 
the  start  and  imderstand  user  limitations; 

•  Ascertain  that  sufficient  time  and  test 
articles  are  planned.  When  the  technol¬ 
ogy  is  stressed,  the  higher  risks  require 
more  test  articles  and  time; 

•  In  general,  parts,  subsystems  and  sys¬ 
tems  should  be  proven  in  that  order  be¬ 
fore  incorporating  them  into  the  next 
higher  assembly  for  more  complete  tests. 
The  instrumentation  should  be  planned 
to  permit  diagnosis  of  trouble; 

•  Major  tests  should  never  be  repeated 
without  an  analysis  of  failure  and  correc¬ 
tive  action.  Allow  for  delays  of  this  na¬ 
ture. 

26.7  SPECIFIC  WEAPON  SYSTEMS 
TESTING  CHECKLIST 

The  DSB  report  is  the  result  of  the  study  of 
past  major  weapon  systems  acquisitions.  It 
was  hoped  that  this  study  would  enhance 
the  testing  community's  understanding  of 
the  role  that  T&E  has  had  in  identifying 
system  problems  during  the  acquisition 
process.  In  the  foreword  of  the  DSB  study, 
the  authors  made  this  statement  about  in¬ 
cluding  the  obvious  testing  activity  in  their 
checklist: 

The  T&E  expert  in  reading  this  volume 
will  find  many  precepts  which  will  strike 
him  as  of  this  type.  These  items  are  in¬ 
cluded  because  examples  were  found 
where  even  the  obvious  has  been  ne¬ 
glected,  not  because  of  incompetence  or 
lack  of  personal  dedication  by  the  people 
in  charge  of  the  program,  but  because  of 
financial  and  temporal  pressures  which 
forced  competent  managers  to  compro¬ 


mise  on  their  principles.  It  is  hoped  that 
the  inclusion  of  the  obvious  will  prevent 
repetition  of  the  serious  errors  which 
have  been  made  in  the  past  when  such 
political,  economical  and  temporal  pres¬ 
sures  have  forced  project  managers  to 
depart  from  the  rules  of  sound  engineer¬ 
ing  practices.. ..In  the  long  run,  taking 
short  cuts  during  T&E  to  save  time  and 
money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and 
in  a  delay  of  delivery  of  the  correspond¬ 
ing  weapon  systems  to  combatant  forces. 

26.7.1  Aircraft  Systems 

26.7.1.1  Concept  Exploration  and 

Definition  Phase 

•  Test  Progrant/T otal  Costs.  Prior  to  Mile¬ 
stone  I,  all  phases  of  the  aircraft  test 
program  should  be  considered  so  the 
total  costs  and  the  development  sched¬ 
ules  include  consideration  of  all  likely 
activities  in  the  overall  program. 

•  Test  Facilities  and  Instrumentation.  Prior 
to  Milestone  I,  the  test  facilities  and  in¬ 
strumentation  requirements  to  conduct 
tests  should  be  generally  identified  along 
with  a  tentative  schedule  of  test  activi¬ 
ties. 

•  Test  Resources  and  Failures.  Ensure  that 
there  are  adequate  funds,  reasonable 
amounts  of  time,  and  acceptable  num¬ 
bers  of  aircraft  planned  for  the  various 
test  program  phases,  and  that  provisions 
are  made  for  the  occurrence  of  failures. 

•  System  Interfaces.  Consider  all  aircraft 
system  interfaces,  their  test  requirements, 
and  probable  costs  at  the  outset  of  the 
Concept  Exploration  and  Definition 
Phase. 

•  Major  Weapon  Subsystems.  If  the  aircraft 
system  relies  on  the  successful  develop- 
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ment  of  a  specific  and  separately-funded 
major  weapon  (such  as  a  gun  or  missile) 
in  order  to  accomplish  its  primary  mis¬ 
sion,  this  "lajor  subsystem  should  be 
developed  and  tested  concurrently  with, 
or  prior  to,  the  aircraft. 

•  Propulsion  System.  If  the  aircraft  pro¬ 
gram  is  paced  by  the  propulsion  system 
development,  an  early  advanced-devel¬ 
opment  project  for  the  propulsion  may 
be  appropriate  for  a  new  concept. 

•  Operational  Scenario.  A  conceptual  op¬ 
erational  scenario  for  operation  and  use 
of  the  aircraft  should  be  developed  so 
that  general  test  plans  can  be  designed. 
This  should  include  purpose,  roles  and 
missions,  threats,  operating  environ¬ 
ments,  logistics  and  maintenance  and 
basing  characteristics.  The  potential 
range  of  values  on  these  aspects  should 
be  stated. 

•  Evaluation  Criteria.  Develop  evaluation 
criteria  to  be  used  for  selecting  the  final 
aircraft  system  design. 

•  Untried  Elements.  The  aircraft  develop¬ 
ment  program  should  include  conclu¬ 
sive  testing  to  eliminate  uncertainties  of 
the  untried  elements. 

•  Brassboard  Avionics  Tests.  The  use  of 
brassboard  or  modified  existing  hard¬ 
ware  to  "prove"  the  concept  will  work 
should  be  seriously  scrutinized  to  ensure 
that  the  demonstrations  and  tests  are 
applicable. 

•  Nuclear  Weapons  Effects.  The  subject  of 
nuclear  weapons  effects  should  be  ad¬ 
dressed  in  the  test  concept  for  all  aircraft 
weapons  systems  where  operational  suit¬ 
ability  dictates  that  survivable  exposure 
to  nuclear  weapons  effects  is  a  require¬ 
ment. 


26.7.1.2  Demonstration  and 
Validation  Phase 

•  By  the  end  of  the  phase,  T&E  plans  and 
test  criteria  should  be  established  so  there 
is  no  question  on  what  constitutes  a  suc¬ 
cessful  test  and  what  performance  is  re¬ 
quired. 

•  Milestones  and  Goals.  Ensure  an  inte¬ 
grated  system  test  plan  that  preestab¬ 
lishes  milestones  and  goals  for  easy  mea¬ 
surement  of  program  progress  at  a  later 
time. 

•  Operating  Concept  and  Environment.  The 
operational  concept  and  the  environ¬ 
ments  in  which  the  aircraft  will  be  ex¬ 
pected  to  operate  and  be  tested  in  OT&E 
should  be  specified. 

•  Test  Program  Building  Blocks.  In  the 
DEM/VAL  Phase,  demonstrate  that 
high-risk  technology  is  in  hand.  In  plan¬ 
ning  the  full-scale  development  test  pro¬ 
gram,  ensure  that  components  and  sub¬ 
systems  are  adequately  qualified  for  in¬ 
corporation  into  the  system  tests. 

•  Technology  Concepts.  Each  concept  to  be 
used  in  the  aircraft  system  (e.g.,  aerody¬ 
namics,  structures,  propulsion)  should 
be  identified  and  coded  according  to  prior 
application,  before  future  research.  Tests 
for  each  concept  should  be  specified  with 
the  effect  of  failure  identified. 

•  DT&E/OT&E  Plan.  The  aircraft  DT&E/ 
OT&E  test  plan  should  be  reviewed  to 
ensure  it  includes  ground  and  flight  tests 
necessary  to  safely  and  effectively  de¬ 
velop  the  system. 

•  Test  Failures.  The  T&E  plans  should  be 
made  assuming  there  will  be  failures; 
they  are  inevitable. 
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•  Multi-Service  Testing.  When  a  new  air¬ 
craft  development  program  requires  joint 
testing  during  OT&E  and  prior  to  Mile¬ 
stone  II,  the  test  plan  should  include  the 
type  of  tests  and  resources  required  from 
other  activities  and  Services. 

•  Traceability.  The  aircraft  development 
and  test  program  should  be  designed 
and  scheduled  so  if  trouble  arises,  its 
source  can  be  traced  back  through  the  lab 
tests  and  the  analytical  studies. 

•  Competitive  Prototype  Tests.  When  a  com¬ 
petitive  prototype  test  program  using 
test  and  operational  crews  is  employed, 
the  aircraft  should  be  compared  on  the 
basis  of  the  performance  of  critical  mis¬ 
sion. 

•  Prototype  Similarity  to  Development  and 
Production  Aircraft.  A  firm  determination 
should  be  made  of  the  degree  of  similar¬ 
ity  of  the  wiiming  prototype  (in  a  com¬ 
petitive  prototype  program)  to  the  de¬ 
velopment  and  production  aircraft.  Thus, 
test  results  that  are  derh'ed  from  the 
prototype  in  the  interim  period  prior  to 
availability  of  the  engineering  develop¬ 
ment  aircraft  can  be  utilized  effectively. 

•  Prototype  Tests.  The  prototype  aircraft 
test  data  should  be  used  to  determine 
where  emphasis  should  be  placed  in  the 
engineering  development  program. 

•  Inlet/Engine/Nozzle  Match.  The  aircraft 
test  program  should  provide  for  an  early 
and  adequate  inlet/engine/nozzle  match 
through  a  well-planned  test  program, 
and  there  should  be  time  programming 
for  corrections. 

•  Subsystem  Tests.  There  should  be  a  bal¬ 
anced  program  for  the  aircraft  subsystem 
tests. 


•  Propulsion  System.  If  the  aircraft  is  paced 
by  the  propulsion  systems  development, 
an  early  advanced-development  project 
for  the  propulsion  may  be  appropriate 
for  a  new  concept. 

•  Electromagnetic  Interface  (EMI)  Testing. 
Full-scale  aircraft  systems  tests  in  an 
anechoic  chamber  are  desirable  for  some 
aircraft. 

•  Parts  Interchange.  Early  plans  should 
provide  for  tests  where  theoretically  iden¬ 
tical  parts,  particularly  in  avionics,  are 
interchanged  to  ensure  that  the  aircraft 
systems  can  be  maintained  in  readiness. 

•  Human  Factors  Demonstration.  Ensure 
adequate  demonstration  of  human  fac¬ 
tors  is  considered  in  the  test  plan. 

•  Military  Preliminary  Evaluation.  Ad¬ 
equate  resources  should  be  scheduled 
for  the  aircraft  Military  Preliminary 
Evaluation  (MPE)  and  a  positive  pro¬ 
gram  should  exist  for  the  utilization  of 
MPE  information  at  the  time  of  OT&E. 

•  User  Participation.  It  is  imperative  that 
the  operational  command  actively  par¬ 
ticipate  in  the  DT&E  Phase  to  ensure  that 
user  needs  are  represented  in  the  devel¬ 
opment  of  the  system. 

•  Maintenance  and  Training  Publications. 
The  aircraft  development  program 
should  provide  for  concurrent  training 
of  crews  and  preparation  of  draft  techni¬ 
cal  manuals  to  be  used  by  lOT&E  main¬ 
tenance  and  operating  crews. 

•  Researchand  Development  (R&D)  Comple¬ 
tion  Prior  to  lOT&E.  The  testing  plans 
should  ensure  that,  before  an  aircraft 
system  is  subjected  to  lOT&E,  the  sub¬ 
systems  essential  to  the  basic  mission 
have  completed  R&D. 
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26.7.1.3  Engineering  and  Manufacturing 
Development  Phase  (old  FSD) 

•  Test  Design.  Test  programs  should  be 
designed  to  have  a  high  probability  of 
early  identification  of  major  deficiencies 
during  the  DT&E  and  lOT&E. 

•  Data  for  Alternate  Scenarios.  By  careful 
attention  to  testing  techniques,  maximize 
the  utility  of  the  test  data  gathered;  air¬ 
craft  instrumentation;  range  instrumen¬ 
tation;  and  data  collection,  reduction  and 
storage. 

•  Test  Milestones.  Development  programs 
should  be  built  around  testing  milestones, 
not  calendar  dates. 

•  Production  Engineering  Influence  on  R&D 
Hardware.  Encourage  that  production  phi¬ 
losophy  and  production  techniques  be 
brought  to  the  maximum  practicable  ex¬ 
tent  into  an  early  phase  of  the  design 
process  for  R&D  hardware. 

•  Running  Evaluation  of  Tests.  Ensure  that 
running  evaluations  of  test  are  conducted. 
If  it  becomes  clear  that  test  objectives  are 
unattainable  or  additional  samples  will 
not  change  the  test  outcome,  ensure  that 
procedures  are  established  for  terminat¬ 
ing  the  test. 

•  Simulation.  Analysis  and  simulation 
should  be  conducted,  where  practicable, 
before  each  phase  of  development  flight 
testing. 

•  Avionics  Mock-up.  Encourage  use  of  a 
complete  avionics  system  installed  in  a 
mock-up  of  the  appropriate  section  or 
sections  of  the  aircraft. 

•  Escape  Systems  Testing.  Ensure  the  air¬ 
crew  escape  system  is  thoroughly  tested 
with  particular  attention  to  redundant 


features,  such  as  pyrotechnic  firing  chan¬ 
nels. 

•  Structural  Testing.  Ensure  that  fatigue 
testing  is  conducted  on  early  production 
airframes.  Airframe  production  should 
be  held  to  a  low  rate  until  satisfactory 
progress  is  shown  in  these  tests. 

•  Gun  Firing  Tests.  All  forms  of  ordnance, 
especially  those  that  create  gases,  must 
be  fired  from  the  aircraft  for  external 
effects  (blast  and  debris),  internal  effects 
(shock)  and  effects  on  the  propulsion 
(inlet  composition  or  distribution). 

•  Post-Stall  Characteristics.  Special  atten¬ 
tion  is  warranted  on  the  post-stall  test 
plans  for  DT&E  and  OT&E. 

•  Subsystem  Performance  History.  During 
DT&E  and  lOT&E  of  aircraft,  ensure  that 
a  performance  history  of  each  aircraft 
subsystem  is  kept. 

•  Flight  Deficiency  Reporting.  Composi¬ 
tion  of  flight  deficiencies  reporting  by 
aircrews,  particularly  those  pertaining 
to  avionics,  should  be  given  special  at¬ 
tention. 

•  Crew  Limitations.  Ensure  aircrew  limi¬ 
tations  are  included  in  the  tests. 

•  Use  of  Operational  Personnel.  Recom¬ 
mend  experienced  operational  person¬ 
nel  help  in  establishing  measures  of  ef¬ 
fectiveness  and  in  other  operational  test 
planning.  In  conducting  OT&E,  use  typi¬ 
cal  operational  aircrews  and  support 
personnel. 

•  Role  of  the  User.  Ensure  that  users  par¬ 
ticipate  in  the  T&E  phase  so  their  needs 
are  represented  in  the  development  of 
the  system  concept  and  hardware. 
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•  Crew  Fatigue  and  System  Effectiveness.  In 
attack  aircraft  operational  testing  and 
particularly  in  attack  helicopter  tests 
where  vibration  is  a  fatiguing  factor,  as¬ 
certain  that  the  tests  include  a  measure  of 
degradation  over  time. 

•  Time  Constraints  on  Crews.  Detailed  op¬ 
erational  test  plans  should  be  evaluated 
to  determine  that  the  test-imposed  con¬ 
ditions  on  the  crew  do  not  invalidate  the 
applicability  of  the  collected  data. 

•  Complete  Basic  DT&E  before  Starting 
OT&E.  Before  the  weapon  system  is  sub¬ 
jected  to  lOT&E,  all  critical  subsystems 
should  have  completed  basic  DT&E  and 
significant  problems  should  be  solved. 

•  Realism  in  Testing.  Ascertain  that  final 
DT&E  system  tests  and  lOT&E  flight 
tests  are  representative  of  operational 
conditions. 

•  Test  All  Profiles  and  Modes.  Tests  should 
be  conducted  to  evaluate  all  planned 
operational  flight  profiles  and  all  pri¬ 
mary  and  backup,  degraded  operating 
modes. 

•  Update  of  Operational  Test  Plans.  Ensure 
that  operational  test  plans  are  reviewed 
and  updated,  as  needed,  to  make  them 
relevant  to  evolving  concepts. 

•  Conduct  lOT&E  Early.  Ensure  that  op¬ 
erational  suitability  tests  are  planned  to 
attempt  to  identify  operational  deficien¬ 
cies  of  new  systems  quickly  so  fixes  can 
be  developed  and  tested  before  large- 
scale  production. 

•  Missile  Launch  Tests.  Review  the  final 
position  fix  planned  before  launching 
inertial-guided  air-to-surface  missiles. 

•  Mission  Completion  Success  Probability. 
Mission  completion  success  probability 


factors  should  be  used  to  measure 
progress  in  the  aircraft  test  program. 

26.7.1.4  Production  and  Deployment 
Phase 

•  Operational  Test  Realism.  Ascertain  op¬ 
erational  testing  is  conducted  under  re¬ 
alistic  conditions. 

•  Design  FOT&E  for  Less-Than-Optimal 
Condition.  Structure  the  FOT&E  logisti¬ 
cal  support  for  simulated  combat  condi¬ 
tions. 

•  New  Threat.  Be  alert  to  the  need  to 
extend  the  FOT&E  if  a  new  threat  shows 
up. 

•  Certification  of  Ordnance.  Ensure  that 
ordnance  to  be  delivered  by  an  aircraft  is 
certified  for  the  aircraft. 

•  Inadvertent  Influence  of  Test.  The  FOT&E 
plans  should  provide  measures  of  ensur¬ 
ing  that  actions  by  observers  and  um¬ 
pires  do  not  unwittingly  influence  trial 
outcome. 

•  Deficiencies  Discovered  In-Service.  Be 
aware  that  in-Service  operations  of  an 
aircraft  system  will  surface  deficiencies 
which  extensive  FOT &E  probably  would 
not  uncover. 

•  Lead  the  Fleet.  Accelerated  Service  test 
of  a  small  quantity  of  early  production 
aircraft  is  advisable  during  FOT&E 
thereafter. 

26.7.2  Missile  Systems 

26.7.2.1  Concept  Exploration  and 
Definition  Phase 

•  Weapon  System  Interfaces.  Consider  sig¬ 
nificant  weapon  system  interfaces,  their 
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test  requirements  and  probable  costs  at 
the  outset  of  the  Concept  Exploration 
and  Definition  Phase.  Ensure  that  the 
program  plan  assembled  before  Mile¬ 
stone  I  includes  an  understanding  of  the 
basic  test  criteria  and  broad  test  plans  for 
the  whole  program. 

•  N umber  ofTest  Missiles.  Ensure  that  there 
is  sufficient  time  and  a  sufficient  number 
of  test  articles  to  support  the  program 
through  its  various  phases.  Compare  the 
program  requirements  with  past  missile 
programs  of  generic  similarity.  If  there  is 
substantial  difference,  then  adequate  jus¬ 
tification  should  be  provided.  The  DT&E 
period  on  many  programs  has  had  to  be 
extended  as  much  as  50  percent. 

•  Test  and  Evaluation  Gap.  A  T&E  gap  has 
been  experienced  in  some  missile  pro¬ 
grams  between  the  time  when  testing 
with  R&D  hardware  was  completed  and 
the  time  when  follow-on  operational  suit¬ 
ability  testing  was  initiated  with  produc¬ 
tion  hardware. 

•  Feasibility  Tests.  Ensure  experimental 
test  evidence  is  available  to  indicate  the 
feasibility  of  the  concept  and  the  avail¬ 
ability  of  the  technology  for  the  system 
development. 

•  Evaluation  of  Conceptual  and  Validation 
Tests.  Results  of  tests  conducted  during 
the  Concept  Exploration  and  Definition 
and  the  DEM/VAL  Phases,  which  most 
likely  have  been  conducted  as  avionics 
brassboard,  breadboard  or  modified  ex¬ 
isting  hardware,  should  be  evaluated 
with  special  attention. 

•  Multi-Service  Testing  Plans.  When  a  new 
missile  development  program  requires 
multi-Service  testing  during  OT&E,  the 
test  plan  should  include  the  type  of  tests 


and  resources  required  from  other  ac¬ 
tivities  and  Services. 

•  Test  Facilities  and  Instrumentation  Re¬ 
quirements.  Before  Milestone  I,  the  test 
facilities  and  instrumentation  require¬ 
ments  to  conduct  tests  should  be  gener¬ 
ally  identified  along  with  a  tentative 
schedule  of  test  activities. 

26.7.2.2  Demonstration  and  Validation 
Phase 

•  Establish  Test  Criteria.  By  the  end  of  the 
DEM/VAL  phase,  test  criteria  should  be 
established  so  there  is  no  question  on 
what  constitutes  a  successful  test  and 
what  performance  is  expected. 

•  Human  Factors.  Ensure  that  the  test  plan 
includes  adequate  demonstration  of  hu¬ 
man  factors  considerations. 

•  Instrumentation  Diagnostic  Capability  and 
Compatibility.  Instrumentation  design, 
with  adequate  diagnostic  capability  and 
compatibility  in  DT&E  and  lOT&E 
phases,  is  essential. 

•  Provisions  for  Test  Failures.  The  DT&E 
and  OT&E  plans  should  include  provi¬ 
sions  for  the  occurrence  of  failures. 

•  Integrated  Test  Plan.  Ensure  develop¬ 
ment  of  an  integrated  system  test  plan 
that  preestablishes  milestones  and  goals 
for  easy  measurement  of  program 
progress  at  a  later  time. 

•  Test  and  Evaluation  Requirements.  En¬ 
sure  that  the  T&E  program  requirements 
are  firm  before  approving  an  R&D  test 
program.  Many  missile  programs  have 
suffered  severe  cost  impacts  as  a  result  of 
this  deficiency.  The  test  plan  must  in¬ 
clude  provisions  to  adequately  test  those 
portions  of  the  operational  envelope  that 
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stress  the  system  including  backup  and 
degraded  operational  modes. 

•  Personnel  Training  Plans.  Ensure  that 
adequate  training  and  certification  plans 
for  test  personnel  have  been  developed. 

•  Test  and  Engineering  Reporting  Format. 
Include  a  T&E  reporting  format  in  the 
program  plan.  Attention  must  be  given 
to  the  reporting  format  in  order  to  pro¬ 
vide  a  consistent  basis  for  T&E  through¬ 
out  the  program  life  cycle. 

•  Program-to-Program  Cross  Talk.  Encour¬ 
age  program-to-program  T&E  cross  talk. 
Test  and  evaluation  problems  and  their 
solutions,  as  one  program,  provide  a  valu¬ 
able  index  of  lessons  learned  and  tech¬ 
niques  for  problem  resolution  on  other 
programs. 

•  Status  of  T&E  Offices.  Ensure  that  T&E 
offices  reporting  to  the  program  man¬ 
ager  or  director  have  the  same  stature  as 
other  major  elements.  It  is  important  that 
the  T&E  component  of  the  system  pro¬ 
gram  office  has  organizational  status  and 
authority  equal  to  configuration  man¬ 
agement,  program  control,  system  en¬ 
gineering,  etc. 

•  Measurement  of  Actual  Environments. 
Thorough  measurements  should  be  made 
to  define  and  understand  the  actual  envi¬ 
ronment  in  which  the  system  compo¬ 
nents  must  live  during  the  captive,  launch 
v^nd  in-flight  phases. 

•  Thoroughness  of  Laboratory  Testing.  Sig¬ 
nificant  time  and  money  will  be  saved  if 
each  component,  each  subsystem,  and 
the  full  system  are  all  tested  as  thor¬ 
oughly  as  possible  in  the  laboratory. 

•  Contract  Form.  The  contract  form  can  be 
extremely  important  to  the  T&E  aspects. 


In  one  program,  the  contract  gave  the 
contractor  full  authority  to  determine 
the  number  of  test  missiles;  and  in  an¬ 
other,  the  contract  incentive  resulted  in 
the  contractor  concentrating  tests  on  one 
optimum  profile  to  satisfy  the  incentive 
instead  of  developing  the  performance 
throughout  important  areas  of  the  enve¬ 
lope. 

•  Participation  of  Operational  Command.  It 
is  imperative  that  the  operational  com¬ 
mand  actively  participate  in  the  DT&E 
phase  to  ensure  that  user  needs  are  rep¬ 
resented  in  the  development  of  the  sys¬ 
tem. 

26.7.2.3  Engineering  and  Manufacturing 

Development  Phase  (old  FSD) 

•  Production  Philosophy  and  Techniques. 
Encourage  that  production  philosophy 
and  production  techniques  be  brought, 
to  the  maximum  practicable  extent,  into 
an  early  phase  of  the  design  process  for 
R&D  hardware.  There  are  many  missile 
programs  in  which  the  components  were 
not  qualified  until  the  missile  was  well 
into  production. 

•  Operational  Flight  Profiles.  Tests  should 
be  conducted  to  evaluate  all  planned 
operational  flight  profiles  and  all  pri¬ 
mary  and  backup  degraded  operating 
modes. 

•  Failure  Isolation  and  Responsive  Action. 
Does  the  system  test  plan  provide  for 
adequate  instrumentation  so  missile  fail¬ 
ures  can  be  isolated  and  fixed  before  the 
next  flight? 

•  Responsive  Actions  for  Test  Failures.  En¬ 
courage  a  closed-loop  reporting  and  reso¬ 
lution  process,  which  ensures  that  each 
test  failure  at  every  level  is  closed  out  by 
appropriate  action;  i.e.,  redesign,  pro¬ 
curement,  retest,  etc. 
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•  Plan  Tests  of  Whole  System.  Plan  tests  of 
the  whole  system  including  proper  phas¬ 
ing  of  the  platform  and  supporting  gear, 
the  launcher,  the  missile  and  user  par¬ 
ticipation. 

•  Determination  of  Component  Configura¬ 
tion.  Conditions  and  component  configu¬ 
ration  during  development  tests  should 
be  determined  by  the  primary  objectives 
of  that  test.  Whenever  a  nonoperational 
configuration  is  diet  ted  by  early  test 
requirements,  tests  should  not  be  chal¬ 
lenged  by  the  fact  that  configuration  is 
not  operational. 

•  Testing  of  Software.  Test  and  evaluation 
should  ensure  that  software  products 
are  tested  appropriately  during  each 
phase.  Software  often  has  been  devel¬ 
oped  more  as  an  add-on  than  as  an  inte¬ 
gral  part  of  the  overall  system.  Software 
requirements  need  the  same  consider¬ 
ation  as  hardware  requirements  in  the 
DEM/VAL  Phase. 

•  Range  Safety  Dry  Runs.  Ensure  the  test 
plan  includes  adequate  test  program/ 
range  safety  dry  runs.  The  government 
test  ranges  have  to  provide  facilities  to 
safely  test  many  different  projects. 

•  Assemblies/Subsystems  Special  Require¬ 
ments. 

•  Seekers  and  tracking  devices, 

•  Propulsion  subsystems, 

•  Connectors  and  their  related  hard¬ 
ware, 

•  Lanyard  assemblies, 

•  Safeing,  arming,  fuzing  and  other 

ordnance  devices. 


•  Review  of  Air-to-Surface  Missile  (ASM) 
Test  Position  Fixes.  Review  the  final  posi¬ 
tion  fix  planned  before  launching  ASMs. 
There  are  instances  in  which  the  opera¬ 
tional  test  of  air-launched  missiles  uti¬ 
lized  artificial  position  fixes  just  prior  to 
missile  launch. 

•  Operator  Limitations.  Ensure  operator 
limitations  are  included  in  the  tests.  Most 
tactical  missiles,  especially  those  used  in 
close  support,  require  visual  acquisition 
of  the  target  by  the  missile  operator  and  / 
or  an  air/ground  controller. 

•  Test  Simulations  and  Dry  Runs.  Plan  and 
use  test  simulations  and  dry  runs.  Dry 
runs  should  be  conducted  for  each  new 
phase  of  testing.  Simulation  and  other 
laboratory  or  ground  testing  should  be 
conducted  to  predict  the  specific  test 
outcome.  The  “wet  run"  test  should  fi¬ 
nally  be  run  to  verify  the  test  objectives. 
Evaluation  of  the  simulation  vs.  the  ac¬ 
tual  test  results  will  help  to  refine  the 
understanding  of  the  system. 

•  Component  Performance  Records.  Keep 
performance  records  on  components. 
There  are  many  examples  in  missile  pro¬ 
grams  that  have  required  parts  stock 
sweeps  associated  with  flight  failures 
and  component  aging  test  programs. 

•  Tracking  Test  Data.  Ensure  the  test  pro¬ 
gram  tracks  data  in  a  readily  usable  man¬ 
ner.  Reliability  and  performance  evalua¬ 
tions  of  a  missile  system  should  break 
down  the  missile's  activity  into  at  least 
the  following  phases; 

•  Prelaunch  including,  captive  carry 

reliability, 

•  Launch, 

•  In-flight, 
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•  Accuracy/fuzing. 

•  Updating  lOT&E  Planning.  Periodically 
update  MPE  and  lOT&E  planning  dur¬ 
ing  the  early  R&D  phase.  Few  missile 
system  programs  have  had  adequate  user 
participation  with  the  desired  continuity 
of  personnel  to  miiumize  the  problems 
of  transition  from  DT&E  to  OT&E  to 
deployment/utilization. 

•  Instrumentation  Provisions  in  Production 
Missiles.  Encourage  built-in  instrumen¬ 
tation  provisions  in  production  missiles. 

•  Constraints  on  Missile  Operator.  Detailed 
test  plans  should  be  evaluated  to  deter¬ 
mine  that  the  test  imposed  constraints  on 
the  missile  operator  do  not  invalidate  the 
applicability  of  the  data  so  collected. 

•  Problem  Fixes  Before  Production.  Ensure 
that  operational  suitability  tests  identify 
operational  deficiencies  of  new  systems 
quickly  so  fixes  can  be  developed  and 
tested  before  large-scale  production. 

•  Flight  Tests  Representative  of  Operations. 
Ascertain  that  final  DT&E  system  tests 
and  lOT&E  flight  tests  are  representative 
of  operational  flights.  Some  ballistic  mis¬ 
sile  R&E  programs  have  shown  high  suc¬ 
cess  rates  in  R&E  flight  tests;  however, 
when  the  early  production  systems  were 
deployed,  they  exhibited  a  number  of 
unsatisfactory  characteristics  such  as 
poor  alert  reliability  and  poor  operational 
teot-flight  reliability. 

26.7.2.4  Production  and  Deployment 
Phase 

•  System  Interfaces  in  Operational  Test.  En¬ 
sure  the  primary  objective  of  an  opera¬ 
tional  test  is  to  obtain  measurements  on 
the  overall  performance  of  the  weapon 
system  when  it  is  interfaced  with  those 


systems  required  to  Of)erationeilly  use 
the  weapons  system. 

•  Realistic  Conditions  for  Operational  Test¬ 
ing.  Ascertain  operational  testing  is  con¬ 
ducted  under  realistic  combat  conditions. 
This  means  that  the  offense /defense 
battle  needs  to  be  simulated  in  some  way 
before  the  weapon  system  evaluation  can 
be  considered  completed.  Whether  this 
exercise  is  conduct^  within  a  single  Ser¬ 
vice  (as  in  the  test  of  a  surface-to-surface 
antitank  missile  against  tanks)  or  among 
Services  (as  in  the  test  of  an  air-to-surface 
missile  against  tanks  with  antiaircraft 
protection),  the  plans  for  such  testing 
should  be  formulated  as  part  of  the  sys¬ 
tem  development  plan. 

•  Testing  All  Operational  Modes.  Ensure 
the  FOT&E  plan  includes  tests  of  any 
operational  modes  not  previously  tested 
in  lOT&E.  All  launch  modes  including 
degraded,  backup  modes  should  be 
tested  in  the  FOT&E  because  the  soft¬ 
ware  interface  with  the  production  hard¬ 
ware  system  should  be  evaluated  thor¬ 
oughly.  Otherwise,  small,  easy-to-fix 
problems  might  preclude  launch. 

•  Extension  of  the  FOT&E  for  New  Threats. 
Be  alert  to  the  need  to  extend  the  FOT&E 
if  a  new  threat  arises.  Few  missile  pro¬ 
grams  perform  any  kind  of  testing  rent¬ 
able  to  evaluating  system  performance 
against  current  or  new  threats. 

•  "Lead-the-Fleet"  Production  Scheduling. 
Lead-the-Fleet  missile  scheduling  and  tests 
should  be  considered. 

•  Test  Fixes.  Test  fixes  result  from  earlier 
operational  testing.  After  the  lOT&E  that 
identified  problem  areas  in  missiles, 
FOT&E  should  evaluate  these  areas  pri¬ 
marily  to  determine  the  adequacy  of  the 
incorporated  fixes,  particularly  if  the 
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lOT&E  did  not  run  long  enough  to  test 
the  fixes. 

•  FOT&E  Feedback  to  Acceptance  Testing. 
Ensure  that  FOT&E  results  are  quickly 
fed  back  to  influence  early  production 
acceptance  testing.  Production  accep¬ 
tance  testing  is  probably  the  final  means 
the  government  normally  will  have  to 
ensure  the  product  meets  specifications. 
Early  acceptance  testing  could  be  influ¬ 
enced  favorably  by  a  quick  feedback  from 
FOT&E  to  acceptance  testing.  This  is  ex¬ 
emplified  by  a  current  ASM  program 
where  production  has  reached  peak  rates, 
and  the  lOT&E  has  not  been  completed. 

26.7.3  Command  and  Control  Systems 

26.7.3.1  Concept  Exploration  and 

Definition  Phase 

•  Conceptual  Test  Philosophy.  The  T&E 
planners  must  understand  the  nature  of 
command  and  control  (C2)  systems  early 
in  the  Concept  Exploration  and  Defini¬ 
tion  Phase.  In  a  complex  command  and 
control  system,  a  total  systems  concept 
must  be  developed  initially.  Total  sys¬ 
tems  life  cycle  must  be  analyzed  so  the 
necessary  requirement  for  the  design  can 
be  established. 

•  The  Importance  of  Software  Testing.  Testers 
should  recognize  that  software  is  a  pac¬ 
ing  item  in  command  and  control  sys¬ 
tems  development. 

•  Software  Test  Scheduling  -  Contractors' 
Facilities.  Provision  should  be  made  for 
including  software  T&E  during  each  phase 
of  C2  systems'  acquisition.  Availability 
of  contractors'  facilities  should  be  con¬ 
sidered. 

•  Evaluation  of  Exploratory  Development 
Tests.  Care  should  be  exercised  in  evaluat¬ 


ing  results  of  tests  conducted  during  ex¬ 
ploratory  development  of  command  and 
control  systems.  These  tests,  which  most 
likely  have  been  conducted  on 
brassboard,  breadboard  or  modified  ex¬ 
isting  hardware,  should  be  evaluated 
with  special  attention. 

•  Feasibility  Testing  for  Field  Compilers. 
Early  test  planning  should  allow  for  simu¬ 
lating  the  computer  system  to  test  for 
field  use  of  compilers,  where  applicable. 

•  Evaluation  of  Test  Plan  Scheduling.  Mile¬ 
stones  should  be  event-oriented,  not  cal¬ 
endar-oriented. 

•  Type  Personnel  Needs  -  Effects  on  T&E.  A 
mix  of  personnel  with  different  back¬ 
grounds  affecting  T&E  is  required. 

•  Planning  for  Joint-Service  OT&E  Before 
Milestone  I.  Joint-Service  OT&E  (multi- 
Service)  should  be  considered  for  com¬ 
mand  and  control  systems. 

26.7.3.2  Demonstration  and 
Validation  Phase 

•  Test  Prototypes.  In  C2  systems,  proto¬ 
types  must  reasonably  resemble  final 
hardware  configuration  from  a  func¬ 
tional-use  standpoint.  When  high  tech¬ 
nical  risk  is  present,  development  should 
be  structured  around  the  use  of  one  or 
more  test  prototypes  designed  to  prove 
the  system  concept  under  realistic  opera¬ 
tional  conditions  before  proceeding  to 
engineering  development. 

•  Test  Objectives — Critical  Issues.  In  addi¬ 
tion  to  addressing  critical  technical  is¬ 
sues,  T&E  objectives  during  the  Concept 
Demonstration  and  Validation  Phase 
should  address  the  functional  issues  of  a 
C2  system. 
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•  Real-Time  Software  —  Demonstration  of 
"Application  Patches."  Tests  of  real-time 
C2  systems  should  include  demonstra¬ 
tions  of  interfaces  whereby  locally  gen¬ 
erated  application  patches  are  brought 
into  being. 

•  Independent  Software  Test-User  Group. 
An  independent  test-user  software  group 
is  needed  during  early  software  qualifi¬ 
cation  testing. 

•  System  Interfaces.  Critical  attention 
should  be  devoted  to  testing  interfaces 
with  other  C2  systems  and  to  interfaces 
between  subsystems.  Particular  attention 
should  be  devoted  to  interfaces  with  other 
C2  systems  and  to  the  interfaces  between 
sensors  (e.g.,  radar  units),  communica¬ 
tions  systems  (e.g.,  modems)  and  the 
specific  processors  (e.g.,  CPUs).  Inter¬ 
face  with  information  processing  C2  sys¬ 
tems  must  also  address  data-element  and 
code-standardization  problems  if  data  is 
to  be  processed  on-line. 

•  Human  Factors.  In  a  C2  system,  human 
factors  must  be  considered  from  the  ear¬ 
liest  prototype  designs  and  testing  pro¬ 
vided.  Testing  should  be  conducted  to 
determine  the  most  efficient  arrangement 
of  equipment  from  the  human  factor 
standpoint;  e.g.,  displays  should  be  ar¬ 
ranged  for  viewing  from  an  optimum 
angle  whenever  possible;  adequate  ma¬ 
neuvering  room  \  ’itlu  vi  installation  con¬ 
straints  should  be  ^iliowed  considering 
the  number  of  personnel  normally  man¬ 
ning  the  facility;  and  console-mounted 
controls  should  be  designed  and  located 
to  facilitate  operation,  minimize  fatigue 
and  avoid  confusion. 

•  Degraded  Operations  Testing.  When  the 
expected  operational  environment  of  a 
C2  system  suggests  that  the  system  may 
be  operated  under  less-than-finely-tuned 


conditions,  tests  should  be  designed  to 
allow  for  performance  measurements 
under  degraded  conditions. 

•  Test-Bed.  The  use  of  a  test-bed  for  study 
and  experimentation  with  new  C2  sys¬ 
tems  is  needed  early  in  the  Concept  Dem¬ 
onstration  and  Validation  Phase. 

•  Software-Hardware  Interfaces.  The  soft¬ 
ware-hardware  interfaces,  with  all  op¬ 
erational  backup  modes  to  a  new  C2 
system,  should  be  tested  early  in  the 
program. 

•  Reproducible  Tests.  Test  plans  should 
contain  a  method  for  allowing  full-load 
message  input  while  maintaining  repro¬ 
ducible  test  conditions. 

•  Cost-Effectiveness.  Field-test  data  is 
needed  during  the  DEM/VAL  Phase  for 
input  to  cost-effectiveness  analyses  of  C2 
systems. 

26.7.3.3  Engineering  and  Manufacturing 
Development  Phase  (old  FSD) 

•  Acquisition  Strategy.  The  acquisition 
strategy  for  the  system  should: 

•  Allow  sufficient  time  between  the 
plarmed  end  of  demonstration  testing 
and  major  procurement  (as  opposed 
to  limited  procurement)  decisions.  This 
provides  flexibility  for  modifying 
plans,  which  may  be  required  during 
the  test  phases  of  the  program.  For 
instance,  because  insufficient  time  was 
allowed  for  testing  one  recent  C2  sys¬ 
tem,  the  program  and  the  contract  had 
to  be  modified  and  renegotiated; 

•  Be  evaluated  relative  to  constraints 
imposed; 

•  Ensure  that  sufficient  dollars  are 
available,  not  only  to  conduct  the 
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planned  T&E  but  to  allow  for  the  addi¬ 
tional  T&E  that  is  always  required  due 
to  failures,  design  changes,  etc. 

•  Problem  Indications.  It  is  important  to 
establish  an  early  detection  scheme  so 
management  can  determine  when  a  pro¬ 
gram  is  becoming  "ill." 

•  Impact  of  Software  Failures.  Prior  to  any 
production  release,  the  impact  of  soft¬ 
ware  failures  on  overall  system  perfor¬ 
mance  parameters  must  be  considered. 

•  Critical  Issues.  lOT&E  should  provide 
the  answers  to  some  critical  issues  pecu¬ 
liar  to  C2  systems.  Some  critical  issues 
that  lOT &E  of  C2  systems  should  answer 
are: 

•  Is  system  mission  reaction  time  a 
significant  improvement  over  present 
systems? 

•  Is  a  backup  mode  provided  for  use 
when  either  airborne  or  ground  sys¬ 
tem  exhibits  a  failure? 

•  Can  the  system  be  transported  as 
operationally  required  by  organic 
transport?  (Consider  ground,  air  and 
amphibious  requirements.) 

•  Is  there  a  special  requirement  for  site 
preparation?  (For  example,  survey  and 
antenna  siteing.) 

•  Can  the  system  be  erected  and  dis¬ 
mantled  in  times  specified?  Are  these 
times  realistic? 

•  Does  relocation  affect  system  align¬ 
ment? 

•  Does  system  provide  for  operation 
during  maintenance? 


•  Can  on-site  maintenance  be  per¬ 
formed  on  shelterless  subsystems  (e.g., 
radar  units)  during  adverse  weather 
conditions? 

•  Displays.  The  display  subsystems  of  a 
C2  system  should  provide  an  essential 
function  to  the  user.  Displays  are  key 
subsystems  of  a  C2  system.  They  provide 
the  link  that  couples  the  operator  to  the 
rest  of  the  system  and  are,  therefore, 
often  critical  to  its  success. 

•  Pilot  Test.  A  pilot  test  should  be  con¬ 
ducted  before  lOT &E  so  sufficient  time  is 
available  for  necessary  changes. 

•  Publications  and  Manuals.  It  is  impera¬ 
tive  that  all  system  publications  and 
manuals  be  completed,  reviewed  and 
selectively  tested  under  operational  con¬ 
ditions  before  beginning  overall  system 
suitability  testing. 

•  Power  Sources.  Mobile,  prime  power 
sources  are  usually  provided  as  govern¬ 
ment-furnished  equipment  (GFE)  and 
can  be  a  problem  area  in  testing  C2  sys¬ 
tems. 

•  lOT&E  Reliability  Data.  The  lOT&E  can 
provide  valuable  data  on  the  operational 
reliability  of  a  C2  system;  this  data  can¬ 
not  be  obtained  through  DT&E. 

•  Subsystem  Tests.  Every  major  subsystem 
of  a  C2  system  should  have  a  successful 
DT&E  before  beginning  overall  system 
operational  testing. 

•  Communications.  The  C2  systenw  must 
be  tested  in  the  appropriate  electromag¬ 
netic  environment  to  determine  the  per¬ 
formance  of  its  communications  system. 

•  Maintenance.  In  lOT&E,  maintenance 
should  include:  a  measurement  of  the 
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adequacy  of  the  maintenance  levels  and 
the  maintenance  practices;  an  assessment 
of  the  impact  that  the  maintenance  plan 
has  on  the  operational  reliability;  the  ac¬ 
cessibility  of  the  major  components  of 
the  system  for  field  maintenance  (e.g., 
cables  and  connectors  are  installed  to 
facilitate  access);  and  verification  that 
the  software  design  for  maintenance  and 
diagnostic  routines  and  procedures  are 
adequate  and  the  software  can  be  modi¬ 
fied  to  accommodate  functional  changes. 

•  Continuity  of  Operations.  The  lOT&E 
should  provide  for  an  impact  assessment 
of  the  failure  of  any  subsystem  element 
of  a  C2  system  on  overall  mission  effec¬ 
tiveness. 

•  Imitative  Deception.  The  lOT&E  should 
provide  for  tests  to  assess  the  susceptibil¬ 
ity  of  the  data  links  of  a  C2  system  to 
imitative  deception. 

•  Demonstration  of  Procedures.  Test  plans 
should  include  a  procedural  demonstra¬ 
tion  whereby  the  tested  C2  system  works 
in  conjunction  with  other  systems. 

•  Government-Furnished  Equipment  and  Fa¬ 
cilities.  Test  and  evaluation  should  be 
concerned  about  the  availability  of  GEE 
equipment  as  specified  in  the  proposed 
contract. 

•  User  Participation  in  T&E.  The  varying 
needs  of  the  user  for  a  C2  system  make 
participation  in  all  phases  of  T&E  man¬ 
datory. 

26.7.3.4  Production  and 

Deplo3anent  Phase 

•  First  Article  Testing.  The  preproduction, 
first  article  testing  and  evaluation  should 
be  designed  and  conducted  to:  (1)  con¬ 
firm  the  adequacy  of  the  equipment  to 


meet  specified  performance  require¬ 
ments;  (2)  confirm  the  adequacy  of  the 
software  not  only  to  meet  current  user 
needs  but  to  accommodate  changing 
needs;  and  (3)  determine  failure  modes 
and  rates  of  the  total  integrated  system. 
This  activity  should  be  followed  by 
FOT&E. 

•  Test  Planners  and  Evaluators.  Use  the 
lOT&E  personnel  in  the  FOT&E  program. 
The  planners  and  evaluators  for  the 
FOT&E  of  the  production  system  can  do 
a  better  job  if  they  are  involved  initially 
in  planning  and  conducting  the  lOT&E. 

26.7.4  Ship  Systems 

26.7.4.1  Concept  Exploration  and 
Definition  Phase 

•  Test  and  Evaluation  Master  Plan.  Prior  to 
Milestone  I,  sufficient  materiel  should  be 
generated  to  allow  for  evaluating  the 
overall  T&E  program. 

•  Test  Objectives  and  Critical  Issues.  In 
evaluating  the  initial  test  concept,  it  is 
important  that  the  test  objectives  during 
the  time  from  Milestone  I  to  Milestone  II 
address  the  major  critical  issues,  espe¬ 
cially  technological  issues. 

•  OT&E  Phasing.  In  evaluating  test  plans, 
look  favorably  on  phasing  where  the 
OT&E  is  run  parallel  to  continued  DT&E. 

•  Test  Facilities  and  Instrumentation  Re¬ 
quired.  Before  Milestone  I,  the  test  facili¬ 
ties  and  instrumentation  requirements 
to  conduct  developmental  and  opera¬ 
tional  tests  and  a  tentative  schedule  of 
test  activities  should  be  identified. 

•  Multiple  Approach  To  Weapon  System 
Development.  Whenever  possible,  the 
weapon  system  concept  should  not  be 
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predicated  on  the  successful  develop¬ 
ment  of  a  single  hardware  or  software 
approach  in  the  various  critical  sub¬ 
systems  (unless  it  has  been  previously 
demonstrated  adequately). 

•  Comparison  of  New  vs.  Old  System.  The 
procedure  for  examining  the  relative  per¬ 
formance  of  new  or  modified  systems  vs. 
old  should  be  indicated  in  the  T&E  plan. 

•  Test  Support  Facilities.  The  phasing  of 
test  support  facilities  must  be  planned 
carefully,  with  some  schedule  flexibility 
to  cover  late  delivery  and  other  unfore¬ 
seen  problems. 

•  Fleet  Operating  Force  Requirements.  The 
requirement  for  fleet  operating  forces  for 
DT&E  or  OT&E  should  be  assessed  early 
in  the  program  and  a  specific  commit¬ 
ment  made  as  to  the  types  of  units  to  be 
employed. 

•  Mission-Related  Measures  of  Effective¬ 
ness.  During  the  Concept  Exploration  and 
Definition  Phase  of  the  acquisition  of  a 
new  class  of  ship,  a  study  effort  should 
be  conunenced  jointly  by  the  Chief  of 
Naval  Operations  (CNO)  and  the  Com¬ 
mander,  Operational  Test  and  Evalua¬ 
tion  Force  (COMOPTEVFOR).  This  ef¬ 
fort  is  to  establish  mission-related  mea¬ 
sures  of  effectiveness,  which  may  be  ex¬ 
pressed  in  numerical  fashion  and  may 
later  be  made  the  subject  of  OT&E  to 
determine  how  closely  the  new  ship  sys¬ 
tem  meets  the  operational  need  for  which 
it  was  conceived. 

•  Ship  T&E  Management.  The  manage¬ 
ment  of  ship  T&E  should  ensure  that  test 
requirements  are  necessary  and  consis¬ 
tent  relative  to  systems/subsystem  as¬ 
pects  and  that  the  necessary  testing  is 
coordm  ated  so  that  test  redundancy  does 
not  become  a  problem. 


•T&E  of  Large,  Integrally-Constructed  Sys¬ 
tems.  Major  subsystems  should  be  proven 
feasible  before  firm  commitment  to  a 
detailed  hull  design. 

26.7.4.2  Demonstration  and 

Validation  Phase 

•  Authentication  of  Human  Factors  Con¬ 
cepts.  Test  and  evaluation  should  authen¬ 
ticate  the  human  factors  concepts  em¬ 
bodied  in  the  proposed  systems  design, 
examining  questions  of  safety,  comfort, 
appropriateness  of  man-machine  inter¬ 
faces,  as  well  as  the  numbers  and  skill 
levels  of  the  personnel  required. 

•  Acquisition  Strategy.  The  acquisition 
strategy  for  a  ship  and  its  subsystenrs 
should  allow  sufficient  time  between  the 
planned  end  of  demonstration  testing 
and  major  procurement  decisions  of  gov¬ 
ernment-furnished  equipment  for  flex¬ 
ibility  to  modify  plans  (may  be  required 
during  the  test  phases  of  the  program). 

•  Evaluation  of  Results  of  Exploratory  Test¬ 
ing.  Results  of  tests  conducted  during 
exploratory  development  and  most  likely 
conducted  on  brassboards,  breadboards 
or  modified  existing  hardware  should  be 
evaluated  carefully. 

•  Software  Testing.  In  view  of  increased 
dependence  upon  computers  in  ship 
management  and  tactical  operation,  soft¬ 
ware  testing  must  be  exceptionally  thor¬ 
ough,  and  integrated  software  testing 
must  begin  as  early  as  possible. 

•  New  Hull  Forms.  When  a  new  type  of 
ship  involves  a  radical  departure  from 
the  conventional  hull  form,  extensive  pro¬ 
totype  testing  should  be  required  prior 
to  further  commitment  to  the  new  hull 
form. 
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•  Effects  of  Hull  and  Propulsion  on  Mission 
Capability.  The  predicted  effects  of  the 
proven  hull  and  propulsion  system  de¬ 
sign  on  the  performance  of  the  ship's 
critical  system  should  be  determined. 

•  Advances  in  Propulsion.  Demonstration 
of  the  use  of  new  propulsion  systems 
should  be  conducted  prior  to  making  the 
decision  to  commit  the  propulsion  sys¬ 
tems  to  the  ship  in  question. 

•  Propulsion  Systems  in  Other  Classes. 
When  an  engine  to  be  used  in  the  propul¬ 
sion  system  of  a  new  ship  is  already 
performing  satisfactorily  in  another  ship, 
this  is  not  to  be  taken  as  an  indication  that 
shortcuts  can  be  taken  in  propulsion  sys¬ 
tem  DT&E,  or  that  no  problems  will  be 
encountered. 

•  The  OT&E  of  Shipboard  Gun  Systems. 
Operational  tests  of  shipboard  gun  sys¬ 
tems  should  simulate  the  stress,  expo¬ 
sure  time  and  other  conditions  of  battle 
so  that  the  suitability  of  the  weapon  can 
be  evaluated  in  total. 

•  Targets  for  Antiaircraft  Warfare  (AAW) 
OT&E.  Operational  test  of  shipboard 
AAW  weapons  demands  the  use  of  tar¬ 
gets  which  realistically  simulate  the  present- 
day  threat. 

•  Waivers  to  T&E  of  Ship  Systems.  Waivers 
to  T&E  of  preproduction  models  of  a 
system  in  order  to  speed  up  production 
and  delivery  should  be  made  only  after 
considering  all  costs  and  benefits  of  the 
waiver,  including  those  not  associated 
with  the  contract. 

•  Environment  Effects  on  Sonar  Domes.  En¬ 
vironmental  effects  on  sonar  domes  and 
their  self-noise  should  be  tested  and 
evaluated  before  the  domes  are  accepted 
as  part  of  the  sonar  system. 


•  Hull/Machinery  Testing  by  Computer 
Simulation.  In  DT&E  ships,  there  will  be 
cases  where  the  best  means  to  conduct 
evaluations  of  particular  hull  and  ma¬ 
chinery  capabilities  is  through  dynamic 
analysis  using  computer  simulation,  with 
later  validation  of  the  simulation  by  ac¬ 
tual  test. 

•  Operational  Reliability.  The  OT&E  should 
provide  valuable  data  on  the  operational 
reliability  of  ship  weapon  systems  that 
cannot  be  obtained  through  DT&E. 

26.7.4.3  Engineering  and  Manufacturing 
Development  Phase  (old  PSD) 

•  Initial  or  Pilot  Phase  of  lOT&E.  Before 
any  operational  tests  to  demonstrate  op¬ 
erational  suitability  and  effectiveness  are 
conducted,  an  initial  or  pilot  test  should 
be  conducted. 

•  Identify  Critical  Subsystems.  In  planning 
for  the  lOT&E  of  a  ship  system,  the  criti¬ 
cal  subsystems,  with  respect  to  mission 
performance,  should  be  identified. 

•  Reliability  of  Critical  Systems.  Test  and 
evaluation  should  determine  the  ex¬ 
pected  reliability  at  sea  of  systems  criti¬ 
cal  to  the  ship's  mobility  and  to  the  pri¬ 
mary  and  major  secondary  tasks. 

•  Consistency  in  Test  Objectives.  There  are 
various  phases  in  testing  a  ship  system. 
One  should  ensure  the  objectives  of  one 
phase  are  not  inconsistent  with  the  objec¬ 
tives  of  the  other  phases. 

•  Single  Screw  Ships.  Test  and  evaluation 
of  the  propulsion  systems  of  ships  with  a 
single  screw  should  be  especially  rigor¬ 
ous  to  determine  failure  rates,  mainte¬ 
nance  and  repair  alternatives. 

•  Problems  Associated  With  New  Hulls. 
Whenever  a  new  hull  is  incorporated 
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into  ship  design,  a  T&E  of  this  hull  should 
be  conducted  prior  to  the  full-rate  pro¬ 
duction  and  incorporation  of  the  major 
weapons  subsystems. 

26.7.4.4  Production  and  Deployment 
Phase 

•  Design  of  Ship  FOT&E.  In  the  testing 
program  of  a  ship  system,  it  should  be 
recognized  that,  although  it  may  be  des¬ 
ignated  as  a  special-purpose  ship,  in  most 
cases  it  will  be  used  in  a  general-purpose 
role  as  well. 

•  Operational  Testing  During  Shakedown 
Periods.  The  time  period  for  FOT&E  of  a 
ship  can  be  used  more  efficiently  if  full 
advantage  is  taken  of  the  periods  imme¬ 
diately  after  the  ship  is  delivered  to  the 
Navy. 

•  Fleet  Operations  in  FOT&E.  A  great  deal 
of  information  on  the  operational  effec¬ 
tiveness  of  a  ship  can  be  obtained  from 
standard  fleet  operations  through  well- 
designed  information  collection,  process¬ 
ing  and  analysis  procedures. 

•  Ship  Antisubmarine  Warfare  (ASW) 
FOT&E  Planning.  In  planning  FOT&E  of 
shipboard  systems,  it  is  important  to  rec¬ 
ognize  the  difficulty  of  achieving  real¬ 
ism,  perhaps  more  so  than  in  other  areas 
of  naval  warfare. 

•  Variable  Depth  Sonar  FOT&E.  The  be¬ 
havior  of  towed  bodies  of  variable  depth 
sonar  systems  and  towed  arrays  should 
be  tested  and  evaluated  under  all  ship 
maneuvers  and  speeds  likely  to  be  en¬ 
countered  in  combat. 

•  Ship  Self-Noise  Tests.  The  magnetic  and 
acoustic  signatures  of  a  ship  can  be  tested 
accurately  only  after  it  is  completed. 

•  Ejfect  of  Major  Electronic  Countermea¬ 
sures  (ECMs)  on  Ship  Capability.  The 


FOT&E  of  a  ship  should  include  tests  of 
the  effectiveness  of  the  ship  when  sub¬ 
jected  to  major  ECM. 

•  Ship  System  Survivability.  Follow-on  Op¬ 
erational  Test  and  Evaluation  of  modem 
ships  should  provide  for  the  assessment 
of  their  ability  to  survive  and  continue  to 
fight  when  subjected  to  battle  damage. 

•  Interlocks.  Shipboard  electronic  systems 
are  designed  with  interlock  switches  that 
open  electrical  circuits  for  safety  reasons 
when  the  equip  ment  cabinets  are  opened. 
The  FOT &E  should  be  able  to  detect  over- 
design  as  well  as  minimum  design  ad¬ 
equacy  of  the  interlock  systems. 

•  Intraship  Communication.  In  conducting 
lead  ship  trials  and  evaluations,  particu¬ 
lar  attention  should  be  given  to  the  op¬ 
erational  impact  resulting  from  absence, 
by  design,  of  intraship  communications 
circuits  and  stations  from  important  op¬ 
erating  locations. 

26.7.5  Surface  Vehicle  Systems 

26.7.5.1  Concept  Exploration  and 

Definition  Phase 

•  Preparing  Test  Plans.  It  is  necessary  that 
a  detailed  evaluation  criteria  be  estab¬ 
lished  that  includes  all  items  to  be  tested. 

•  Validation  Test  Plans.  Prior  to  Milestone 
I,  a  plan  should  be  prepared  for  evaluat¬ 
ing  the  overall  T&E  program.  As  part  of 
this,  a  detailed  T&E  plan  for  those  tests  to 
be  conducted  before  Milestone  II  to  vali¬ 
date  the  concept  and  hardware  approach 
to  the  vehicle  system  should  be  devel¬ 
oped.  The  objective  of  the  validation  test 
plan  is  to  fully  evaluate  the  performance 
characteristics  of  the  new  concept  ve¬ 
hicle.  This  test  plan  cannot  be  developed, 
of  course,  until  the  performance  charac¬ 
teristics  are  defined. 
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•  Performance  Characteristics  Range.  Stated 
performance  characteristics  derived  from 
studies  should  be  measured  early  in  the 
program.  Unrealistic  performance  re¬ 
quirements  can  lead  to  false  starts  and 
costly  delays. 

•  Operating  Degradation.  System  perfor¬ 
mance  degrades  under  field  conditions. 
Anticipated  degradation  must  be  con¬ 
sidered  duringT&E.  When  a  system  must 
operate  at  peak  performance  during  de¬ 
velopment  test/operational  test  (DT /OT) 
to  meet  the  specified  requirements,  it 
will  then  be  likely  to  perform  at  a  lesser 
level  when  operated  in  the  field. 

•  Test  Personnel.  The  test  director  and/or 
key  members  of  the  test  planning  group 
within  the  project  office  should  have  sig¬ 
nificant  T&E  experience. 

•  Design  Reviews.  T&E  factors  and  expe¬ 
rience  must  influence  the  system  design. 
The  application  of  knowledge  derived 
from  past  experience  can  be  a  major  asset 
in  arriving  at  a  sound  system  design. 

•  Surrogate  Vehicles.  When  high  technical 
risk  is  present,  development  should  be 
structured  around  the  use  of  one  or  more 
surrogate  vehicles  designed  to  prove  the 
system  concept  under  realistic  opera¬ 
tional  conditions  before  proceeding  with 
further  development. 

•  Test  Facilities  and  Scheduling.  Before  Mile¬ 
stone  I,  test  range  and  resource  require¬ 
ments  to  conduct  validation  tests  and  a 
tentative  schedule  of  test  activities  should 
be  identified. 

26.7.5.2  Demonstration  and 
Validation  Phase 

•  Vulnerability.  The  vulnerability  of  ve¬ 
hicles  should  be  estimated  on  the  basis  of 
testing. 


•  Gun  and  Ammunition  Performance.  Gun 
and  ammimition  develc^ment  should  be 
considered  a  part  of  overall  tank  system 
development.  When  a  new  gun  tube,  or 
one  which  has  not  been  mounted  previ¬ 
ously  on  a  tank  chassis,  is  being  evalu¬ 
ated,  all  ammunition  types  (including 
missiles)  planned  for  use  in  that  system 
should  be  test  fired  under  simulated 
operational  conditions. 

•  Increased  Complexity.  The  addition  of 
new  capabilities  to  an  existing  system  or 
system  type  will  generally  increase  com¬ 
plexity  of  the  system  and,  therefore,  in¬ 
crease  the  types  and  amount  of  testing 
required  and  the  time  to  perform  these  tests. 

•  Component  Interfaces.  Prior  to  assembly 
in  a  prototype  system,  component  sub¬ 
systems  should  be  assembled  in  a  mock- 
up  and  verified  for  physical  fit,  human 
factors  considerations,  interface  compat¬ 
ibility  and  for  electrical  and  mechanical 
compatibility. 

•  Determining  Test  Conditions.  During 
validation,  test  conditions  should  be  de¬ 
termined  by  the  primary  objectives  of 
that  test  rather  than  by  more  general 
considerations  of  realism. 

•  Test  Plan  Development.  The  test  plan 
developed  by  this  point  should  be  in  nearly 
final  form  and  include,  as  a  minimum; 

•  A  description  of  requirements, 

•  The  facilities  needed  to  make  evalu¬ 
ations, 

•  The  schedule  of  evaluations  and  fa¬ 
cilities, 

•  The  reporting  procedure,  the  objec¬ 
tive  being  to  communicate  test  results 
in  an  understandable  format  to  all  pro¬ 
gram  echelons. 
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•  The  T&E  guidelines,  and 

•  A  further  refinement  of  the  cost  esti¬ 
mates  which  were  initiated  during  the 

Conceptual  Phase. 

•  Demonstration  Tests.  Demonstration 
tests  should  show  satisfactory  meeting 
of  success  criteria  which  are  meaningful 
in  terms  of  operational  usage.  It  is  essen¬ 
tial  in  designing  contractually  required 
demonstration  tests,  upon  whose  out¬ 
come  large  incentive  payments  or  even 
program  continuation  may  depend,  to 
specify  broader  success  criteria  than  sim¬ 
ply  hit  or  miss  in  a  single  given  scenario. 

•  Reliability  Testing.  Reliability  testing 
should  be  performed  on  component  and 
subsystem  assemblies  before  testing  of 
the  complete  vehicle  system.  Prior  to  full 
system  testing,  viable  component  and 
subsystem  tests  should  be  conducted. 

•  Human  Factors.  In  evaluating  ground 
vehicles,  human  factors  should  be  con¬ 
sidered  at  all  stages  starting  with  the 
design  of  the  prototype. 

•  Test  Plan  Scheduling.  Test  plan  schedul¬ 
ing  should  be  tied  to  event  milestones 
rather  than  to  the  calendar.  In  evaluating 
the  adequacy  of  the  scheduling  as  given 
by  test  plans,  it  is  important  that  mile¬ 
stones  be  tied  to  the  major  events  of  the 
weapon  system  (meeting  stated  require- 
mei  ts)  and  not  the  calendar. 

•  Test  Failures.  The  T&E  schedule  should 
be  sufficiently  flexible  to  accommodate 
failures  and  correction  of  identified  prob¬ 
lems. 

26.7.5.3  Engineering  and  Manufacturing 

Development  Phase  (old  PSD) 

•  Planning  the  lOT&E.  The  lOT&E  should 
be  cost-effective  and  provide  meaning¬ 
ful  results. 


•  Pilot  and  Dry-Run  Tests.  A  scheduled 
series  of  tests  should  be  preceded  by  a 
dry  run,  which  verifies  that  the  desired 
data  will  be  obtained. 

•  Comparison  Testing.  The  test  program 
should  include  a  detailed  comparison  of 
the  characteristics  of  a  new  vehicle  sys¬ 
tem  with  those  of  existing  systems,  alter¬ 
nate  vehicle  system  concepts  (if  appli¬ 
cable)  and  those  of  any  system(s)  being 
replaced. 

•  Simulation.  Simulation  techniques  and 
equipment  should  be  utilized  to  enhance 
data  collection.  Creation  of  histograms 
for  each  test  course  provides  a  record  of 
conditions  experienced  by  the  vehicle 
during  testing.  Use  of  a  chassis  dyna¬ 
mometer  can  produce  additional  drivel¬ 
ing  endurance  testing  with  more  com¬ 
plete  instrumentation  coverage. 

•  Environmental  Testing.  Ground  vehicles 
should  be  tested  in  environmental  con¬ 
ditions  and  situations  comparable  to 
those  in  which  they  will  be  expected  to 
perform. 

•  System  Vulnerability.  For  combat  ve¬ 
hicles,  some  estimate  of  vulnerability  to 
battle  damage  should  be  made. 

•  Design  Criteria  Verification.  Subsystem 
design  criteria  should  be  compared  with 
actual  characteristics. 

•  Electromagnetic  Testing.  Vehicle  testing 
should  include  electromagnetic  testing. 

•  System  Strength  Testing.  In  evaluating 
ground  vehicles,  early  testing  should 
verify  intrinsic  strength.  This  implies  op¬ 
eration  with  maximum  anticipated  load¬ 
ing,  including  trailed  loads  at  maximum 
speeds  and  over  worst-case  grades,  sec¬ 
ondary  roads  and  cross-country  condi¬ 
tions  for  which  the  vehicle  was  devel¬ 
oped  or  procured.  This  test  is  intended  to 
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identify  deficient  areas  of  design,  not  to 
break  the  machinery. 

•  Component  Compatibility.  Component 
compatibility  should  be  checked  through 
the  duration  of  the  test  sequence. 

•  Human  Interface.  Critiques  of  good  and 
bad  features  of  the  vehicle  should  be 
made  early  in  the  prototype  stage  while 
adequate  time  remains  to  make  any  indi¬ 
cated  changes. 

•  Serviceability  Testing.  Ground  vehicles 
should  be  tested  and  evaluated  to  deter¬ 
mine  the  relative  ease  of  serviceability, 
particularly  with  high-frequency  opera¬ 
tions. 

•  Experienced  User  Critique.  Ground  ve¬ 
hicle  user  opinions  should  be  obtained 
early  in  the  development  program. 

•  Troubleshooting  DuringTests.  Provisions 
should  be  made  to  identify  subsystem 


failure  causes.  Subsystems  may  exhibit 
failures  during  testing.  Adequate  provi¬ 
sions  should  be  made  to  permit  trouble¬ 
shooting  and  identification  of  defective 
components  and  inadequate  design. 

26.7.5.4  Production  and  Deployment 
Phase 

•  Performance  and  Reliability  Testing.  The 
production  first-article  testing  should  verify 
the  performance  of  the  vehicle  system  and 
determine  the  degradation,  failure  modes 
and  failure  rates. 

•  Lead-tlie-Fleet  Testing.  At  least  one  pro¬ 
duction  prototype  or  initial  production 
model  vehicle  should  be  allocated  to  inten¬ 
sive  testing  to  accumulate  high  operating 
time  in  a  short  period. 

•  User  Evaluation.  User-reported  shortcom¬ 
ings  should  be  followed  up  to  determine 
problem  areas  requiring  correction.  Fixes 
should  be  evaluated  during  an  FOT&E. 
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APPENDIX  A 

ABBREVIATIONS  AND  ACRONYMS 


AAE 

Army  Acquisition  Executive 

AAH 

Advanced  Attack  Helicopter 

ACM 

Advanced  Cruise  Missile 

ADATS 

Army  Development  and  Acquisition  Threat  Simulators 

ADM 

Acquisition  Decision  Memorandum 

AFEWES 

Air  Force  Electronic  Warfare  Evaluation  Simulator 

AFMC 

Air  Force  Materiel  Command 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center 

AF/TE 

Air  Force/Test  and  Evaluation  Office 

ALCM 

Air  Launch  Cruise  Missile 

AMC 

Army  Materiel  Command 

AMARC 

Army  Materiel  Acquisition  Review  Committee 

APB 

Acquisition  Program  Baseline 

ASAF(A) 

Asst.  Secretary  of  Air  Force  (Acquisition) 

ASA  (RD&A) 

Asst.  Sec.  of  Army  (Research,  Dev.  and  Acquisition) 

ASD(PAE) 

Asst.  Sec.  of  Def.  (Program  Analysis  and  Evaluation) 

ASN  (RD&A) 

Asst.  Sec.  of  Navy  (Research,  Dev.  and  Acquisition) 

ATD 

Advanced  Technology  Demonstration 

ATE 

Automatic  Test  Equipment 

AWACS 

Airborne  Warning  and  Control  System 

BIS 

Board  of  Inspection  and  Survey 

BIT 

Built-In  Test 

BLRIP 

Beyond  LRIP 

C2 

Command  and  Control 

C3 

Command,  Control  and  Communication 

C3I 

Command,  Control,  Communications,  Intelligence 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

CDS 

Congressional  Data  Sheets 

CE 

Concept  Exploration 

CED 

Concept  Exploration /Definition  I  h.-'se 

CEP 

Circle  Error  Probability 

CLIN 

Contract  Line  Item  Number 

CNO 

Chief  of  Naval  Operations 

CNP 

Candidate  Nomination  Proposal 

COCI 

Critical  Operational  Issues  and  Criteria 

COEA 

Cost  and  Operational  Effectiveness  Analysis 

COI 

Critical  Operational  Issue 

C.  »;  OPTEVFOR 

Commander,  Operational  Test  and  Evaluation  Force 

CSC 

Computer  Software  Component 

CSCI 

Computer  Software  Configuration  Item 

CSTA 

Combined  Systems  Test  Activity 

CSU 

Computer  Software  Unit 

DA 

Developing  Agency  (Navy) 

DAB 

Defense  Acquisition  Board 

DAE 

Defense  Acquisition  Executive 

DEM/VAL 

Demonstration/Validation  Phase 

DAG 

Data  Automation  Group 

DBDD 

Data  Base  Design  Document 

DCP 

Decision  Coordination  Paper 

DDDR&E 

Deputy  Director  Defense  Research  and  Engineering 

DEMVAL 

Demonstration  and  Validation  Phase 

DID 

Data  Item  Description 

DLT 

Design  Limit  Test 

DMSO 

Defense  Modeling  and  Simulation  Office 

DOD 

Department  of  Defense 

DODD 

Department  of  Defense  Directive 

DODI 

Department  of  Defense  Instruction 

DOE 

Department  of  Energy 

DOT&E 

Director,  Operational  Test  and  Evaluation 

DPESO 

DOD  Product  Engineering  Services  Office 

DPML 

Deputy  Program  Manager,  Logistics 

DPRB 

Defense  Planning  and  Resources  Board 

DPRO 

Defense  Plant  Representative  Office 

DSARC 

Defense  Systems  Acquisition  Review  Coimcil  (now  DAB) 

DSB 

Defense  Science  Board 

DT 

Development  Test 

DT&E 

Development  Test  and  Evaluation 

DTE 

Director,  Test  and  Evaluation 

DTESG 

Defense  Test  and  Evaluation  Steering  Group 

DUSA(OR) 

Deputy  Undersecretary  of  Army  (Operations  Research) 

DV 

Demonstration/Validation  Phase 

DVAL 

Data  Link  Vulnerability  Analysis 

EA 

Evolutionary  Acquisition 

EC 

Electronic  Combat 

ECM 

Electronic  Countermeasures 

ECCM 

Electronic  Counter-Countermeasures 

ECM 

Electronic  Countermeasures 

ECR 

Engineering  Change  Review 

EDT 

Engineering  Design  Test 

EMD 

Engineering  and  Manufacturing  Phase 

EMI 

Electromagnetic  Interference 

EMP 

Electromagnetic  Pulse 

EOA 

Early  Operational  Assessment 

ERAM 

Extended  Range  Anti-armor  Munitions 

ESM 

Electronic  Support  Measures 

ESS 

Environmental  Stress  Screening 

EW 

Electronic  Warfare 

FAADS 

Forward  Area  Air  Defense  System 

FAT 

First  Article  Test 

FCA 

Functional  Configuration  Audit 

FDT&E 

Force  Development  Tests  and  Experimentation 

FORSCOM 

Forces  Command 

FOT&E 

Follow-on  Operational  Test  and  Evaluation 

FQR 

Formal  Qualification  Review 

FSD 

Full  Scale  Development  (now  EMD) 

FWE 

Foreign  Weapons  Evaluation 

FYTP 

Five  Year  Test  Program 

GPMO 

Government  Program  Management  Office 

HWCI 

Hardware  Configuration  Item 

HWIL 

Hardware-in-the-Loop 

ICBM 

Intercontinental  Ballistic  Missile 

IDD 

Interface  Decision  Document 

lEP 

Independent  Evaluation  Plan 

IFFN 

Identification,  Friend,  Foe,  Neutral 

IFPP 

Information  for  Proposal  Preparation 

ILS 

Integrated  Logistics  Support 

ILSMT 

Integrated  Logistic  Support  Management  Team 

ILSP 

Integrated  Logistics  Support  Plan 

IOC 

Initial  Operating  Capability 

lOT&E 

Initial  Operational  Test  and  Evaluation 

IPS 

Integrated  Program  Summary 

IRA 

Industrial  Resource  Analysis 

IRS 

Interface  Requirements  Specification 

ITEA 

International  Test  and  Evaluation  Association 

HP 

Integrated  Resource  Analysis 

IV&V 

Independent  Verification  and  Validation 

JCG(T&E) 

Joint  Commanders  Group  (T&E) 

JLF 

Joint  Live  Fire 

JRD 

Joint  Requirements  Document 

JROC 

Joint  Requirements  Oversight  Council 

JT&E 

Joint  Test  and  Evaluation 

JTC3A 

Joint  Tactical  C3  Agency 

LFT 

Live  Fire  Test 

LFT&E 

Live  Fire  Test  and  Evaluation 

LRIP 

Low-Rate  Initial  Production 

LSA 

Logistics  Support  Analysis 

LSAR 

Logistics  Support  Analysis  Report 

MAA 

Mission  Area  Analysis 

MAJCOM 

Major  Commands 

MCCR 

Mission  Critical  Computer  Resources 

MCOTEA 

Marine  Corps  Operational  Test  and  Evaluation  Agency 

MIL-SPEC 

Military  Specification 

MIL-STD 

Military  Standard 

MMOU 

Multinational  Memorandum  of  Understanding 

MNS 

Mission  Needs  Statement 

MOE 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 
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MOU 

MPE 

MRTFB 

MS 

MSTIRC 

NAPMA 

NAVAIR 

NAVSEA 

NBC 

NDCP 

NDI 

OJCS 

NH&S 

O&M 

O&S 

OA 

OMB 

OPEVAL 

OPNAV 

OPNAVIST 

OPSEC 

OPTEC 

OPTEVFOR 

ORD 

ORMAS/TE 

OSD 

OT 

OT&E 

OTA 

OTD 

OTEA 

OTO 

OTP 

P3I 

PAT&E 

PCA 

PCO 

PDR 

PDSS 

PEP 

PM 

PMO 

PO 

POM 

PPBS 

PPQT 

PQT 


Memorandum  of  Understanding 
Military  Preliminary  Evaluation 
Major  Range  and  Test  Facility  Base 
Milestone 

Multi-Service  Test  Livestment  Resource  Committee 
North  Atlantic  Program  Management  Agency 
Naval  Air  Systems  Command 
Naval  Sea  Systems  Command 
Nuclear,  Biological,  and  Chemical 
Navy  Decision  Coordinating  Paper 
Nondevelopment  Item 
Office  of  the  Joint  Chiefs  of  Staff 
Nuclear  Hardness  and  Survivability 
Operations  and  Maintenance 
Operations  and  Support 
Operational  Assessment 
Office  of  Management  and  Budget 
Operational  Evaluation 
Operational  Navy 
Operational  Navy  Instruction 
Operations  Security 

Operational  Test  and  Evaluation  Command 

Operational  Test  and  Evaluation  Force 

Operational  Requirement  Document 

Operational  Resource  Mgmt  Assessment  Systems  for  T&E 

Office  of  the  Secretary  of  Defense 

Operational  Test 

Operational  Test  and  Evaluation 

Operational  Test  Agency 

Operational  Test  Director 

Operational  Test  and  Evaluation  Agency 

Operational  Test  Organization 

Outline  Test  Plan 

Preplaimed  Product  Improvements 
Production  Acceptance  Test  and  Evaluation 
Physical  Configuration  Audit 
Primary  Contracting  Officer 
Preliminary  Design  Review 
Post-Deployment  Software  Support 
Producibility  Engineering  Plan 
Program  Manager 
Program  Management  Office 
Program  Office,  Purchase  Order 
Program  Objectives  Memorandum 
Planning,  Programming,  and  Budgeting  System 
Preproduction  Qualification  Tests 
Production  Qualification  Test 
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PRAT 

Production  Reliability  Acceptance  Test 

PRESINSURV 

President  of  the  Boards  of  Inspection  and  Survey 

PRR 

Production  Readiness  Review 

QOT&E 

Qualification  Operational  Test  and  Evaluation 

R&E 

Research  and  Engineering 

RAM 

Reliability,  Availability,  and  Maintainability 

R&D 

Research  and  Development 

RAS 

Requirements  Allocations  Sheet 

RCS 

Radar  Cross  Section 

RDT 

Reliability  Development  Testing 

RDT&E 

Research,  Development,  Test  and  Evaluation 

REP 

Request  for  Proposal 

RGT 

Reliability  Growth  Test 

RM 

Resource  Manager 

RQT 

Reliability  Qualification  Test 

SAR 

Selected  Acquisition  Report 

SDD 

Software  Design  Document 

SDI 

Strategic  Defense  Initiative 

SDIO 

Strategic  Defense  Initiative  Organization 

SDP 

Software  Development  Plan 

SDR 

System  Design  Paper 

SECARMY 

Secretary  of  the  Army 

SECDEF 

Secretary  of  the  Defense 

SECNAV 

Secretary  of  the  Navy 

SEE 

Stability  Enhancement  Function 

SEMP 

System  Engineering  Management  Plan 

SEMS 

System  Engineering  Management  Schedule 

SIL 

Software  Integration  Laboratory 

SIS 

Stall  Inhibit  System 

SON 

Statement  of  Operational  Need 

SOW 

Statement  of  Work 

SPAWAR 

Space  and  Warfare 

SPEC 

Specification 

SPO 

System  Program  Office 

SRR 

Systems  Requirements  Review 

SRS 

Software  Requirement  Specification 

SSD 

Segment  Design  Document 

SSR 

Software  Specification  Review 

STAR 

System  Threat  Assessment  Report 

STP 

Software  Test  Plan 

SQA 

Software  Quality  Assurance 

SW 

Software 

T&E 

Test  and  Evaluation 

TAAF 

Test,  Analyze  and  Fix 

TADS 

Theater  Air  Defense  System 

TAFT 

Test,  Analyze,  Fix  and  Test 

TEAM 

Test,  Evaluation,  Analysis,  and  Modeling 
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TEC 

Test  and  Evaluation  Committee 

TECG 

T&E  Coordinating  Group 

TECHEVAL 

Technical  Evaluation  (Navy  Term) 

TECOM 

Test  and  Evaluation  Command 

TEMA 

Test  and  Evaluation  Agency 

TEMP 

Test  and  Evaluation  Master  Plan 

TEP 

Test  and  Evaluation  Plan 

TIWG 

Test  Integrated  Working  Group 

TLS 

Time  Line  Sheet 

TM 

Technical  Manual 

TMC 

Test  Management  Council 

TPO 

Test  Program  Outline 

TPM 

Technic^  Performance  Measurement 

TPWG 

Test  Planning  Working  Group 

TR 

Test  Report 

TRADOC 

Training  and  Doctrine  Command 

TRMS 

TRADCDC  Resource  Management  System 

TRR 

Test  Readiness  Review 

TSARC 

Test  Schedule  and  Review  Committee 

USD(A) 

Under  Secretary  of  Defense  for  Acquisition 

WBS 

Work  Breakdown  Structure 

WSMR 

White  Sands  Missile  Range 
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APPENDIX  B 
DOD  GLOSSARY  OF 
TEST  TERMINOLOGY 

ACCEPTANCE  TRIALS  —  Trials  and  material  inspection  conducted  underway  by  the 
trail  board  for  ships  constructed  in  a  private  shipyard,  to  determine  suitability  for 
acceptance  of  a  ship. 

ACCRUED  EXPENDITURES  —  Costs  incurred  during  a  given  period  representing 
liabilities  incurred  for  goods  and  services  received,  other  assets  acquired  and  performance 
accepted,  whether  or  not  pa5anent  has  been  made. 

ACQUISITION  —  The  process  of  planning,  designing,  producing  and  distributing  a 
weapon  system /equipment.  Acquisition  in  this  sense  includes  the  conception,  validation, 
full-scale  development,  production  and  deployment/operational  phases  of  the  weapon 
systems /equipment  project.  For  weapon  systems/equipments  not  being  procured  by  a 
project  manager,  it  encompasses  the  entire  process  from  inception  of  the  requirement 
through  the  operational  phase. 

ACQUISITION  CATEGORY  ( ACAT) — One  of  four  acquisition  categories  established  by 
the  Department  of  Defense  that  governs  acquisition  procedures  and  responsibilities  and 
assigns  respective  decision  authority  levels. 

ACQUISITION  RISK  —  The  change  that  some  elements  of  an  acquisition  program 
produces  an  unintended  result  with  adverse  effect  on  system  effectiveness,  suitability,  cost 
or  availability  for  deployment. 

ADVANCED  DEVELOPMENT  (Budget  Category  6.3) — Includes  all  projects  which  have 
moved  into  the  development  of  hardware  for  test. 

AGENCY  COMPONENT  —  A  major  organizational  subdivision  of  an  agency.  For 
example:  the  Army,  Navy,  Air  Force  and  Defense  Supply  Agency  are  agency  components 
of  the  Department  of  Defense.  The  Federal  Aviation,  Urban  Mass  Transportation  and  the 
Federal  Highway  Administrations  are  agency  components  of  the  Department  of  T ranspor- 
tation. 

ALLOCATION  —  An  authorization  by  a  designated  official  of  a  component  of  the 
Department  of  Defense  making  funds  available  within  a  prescribed  amount  to  an  operating 
agency  for  the  purpose  of  making  allotments;  i.e.,  the  first  subdivision  of  an  apportionment. 

ANALYSIS  —  The  qualitative  and/or  quantified  evaluation  of  information  requiring 
technical  knowledge  and  judgment. 

APPORTIONMENT  —  A  determination  by  the  Office  of  Management  and  Budget  as  to 
the  amoimt  of  obligations  which  may  be  incurred  when  the  nature  of  the  work  involved 
prevents  the  preparation  of  definitive  requirements,  specifications  or  cost  data.  Sometimes 
called  Letter  of  Intent. 
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AUTHORIZATION  —  Basic  substantive  legislation  enacted  by  Congress  which  sets  up  a 
federal  program  or  agency  either  indefinitely  or  for  a  given  period  of  time.  Such  legislation 
sometimes  sets  limits  on  the  amount  that  can  subsequently  be  appropriated,  but  does  not 
usually  provide  budget  authority. 

AUTOMATIC  TEST  EQUIPMENT  (ATE)  —  Equipment  designed  to  automatically 
conduct  analysis  of  functional  or  static  parameters  and  evaluate  the  degree  of  performance 
degradation  and  perform  fault  isolation  of  unit  malfunctions. 

BASELINE,  APPROVED  —  The  combination  of  approved  program  schedule,  configura¬ 
tion,  performance  characteristics,  acquisition,  strategy,  and  other  business  aspects  that 
constitutes  the  variables  reflected  in  either  the  appropriate  acquisition  milestone  approval 
for  that  acquisition  category  or  as  reflected  in  the  latest  approved  program  management 
proposal  action. 

BRASSBOARD  CONFIGURATION  —  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  and  operational  data.  Usually,  it  will 
be  a  model  sufficiently  hardened  for  use  outside  of  laboratory  environments  to  demon¬ 
strate  the  technical  and  operational  principles  of  immediate  interest.  It  may  resemble  the 
end-item  but  is  not  intended  for  use  as  the  end-item. 

BREADBOARD  CONFIGURATION  —  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  develop  technical  data.  Usually,  it  will  be  configured  only 
for  laboratory  use  to  demonstrate  the  technical  principles  of  immediate  interest.  It  may  not 
resemble  the  end-item  and  is  not  intended  for  use  as  the  projected  end-item. 

BUDGET  —  A  planned  program  for  a  fiscal  period  in  terms  of:  (a)  estimated  costs, 
obligations  and  expenditures;  (b)  source  of  funds  for  financing,  including  reimbursements 
anticipated  and  other  resources  to  be  applied;  and  (c)  explanatory  and  workload  data  on 
the  projected  programs  and  activities. 

CONCEPT  EVALUATION  PROGRAM  (CEP)  —  A  specifically-funded  Army  innovative 
testing  program.  Hie  CEPs  provide  commanders  and  combat  developers  a  quick  reaction 
and  simplified  process  to  resolve  combat  development,  doctrinal  and  training  issues.  In 
addition,  CEPs  solidify  combat  development  requirements  and  support  early  milestone 
decisions.  Also,  the  CEP  is  used  to  provide  an  experimental  data  base  for  requirements 
documents  and  to  expedite  the  materiel  acquisition  process;  however,  CEPs  are  not  to  be 
used  as  the  primary  tests  to  support  decision  review  production  decisions.  The  CEP  may 
be  conducted  at  any  time  to  support  the  concept  evaluation  process.  Issues  satisfied  during 
the  conduct  of  a  CEP  need  not  be  examined  during  formal  operational  test  to  minimize 
testing.  Data  from  CEPs  may  be  used  as  another  source  for  preparing  the  independent 
evaluation  report. 

CONTINUOUS  COMPREHENSIVE  EVALUATION  (C2E)  —  A  continuous  process, 
extending  from  concept  definition  through  deployment,  that  evaluates  the  operational 
effectiveness  and  suitability  of  a  system  by  analysis  of  all  available  data. 
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COMBAT  SYSTEM  —  The  equipment,  computer  programs,  people  and  documentation 
organic  to  the  accomplishment  of  the  mission  of  an  aircraft,  surface  ship  or  submarine;  it 
excludes  the  structure,  material,  propulsion,  power  and  auxiliary  equipment,  transmis¬ 
sions  and  propulsion,  fuels  and  control  systems,  and  silencing  inherent  in  the  construction 
and  operation  of  aircraft,  surface  ships  and  submarines. 

CONFIGURATION  MANAGEMENT — A  discipline  applying  technical  and  administra¬ 
tive  direction  and  surveillance  to:  (1)  identify  and  document  the  functional  and  physical 
characteristics  of  a  configuration  item,  (2)  control  changes  to  those  characteristics,  and  (3) 
record  and  report  change  processing  and  implementation  status. 

CONTRACT  —  An  agreement,  enforceable  by  law,  between  two  or  more  competent 
parties  to  do  or  not  do  something  that  is  not  prohibited  by  law  for  a  legal  consideration. 

CONTRACTOR  SUPPORT — An  arrangement  during  initial  development  or  production 
of  end-items  whereby  a  contractor  furnished  required  material  and  maintenance  of  an  end- 
item  or  system,  pending  assumption  of  supply  by  the  military  service. 

COST  AND  OPERATIONAL  EFFECTIVENESS  ANALYSIS  —  An  analysis  of  the 
estimated  costs  and  operational  effectiveness  of  alternative  materiel  systems  to  meet  a 
mission  need  and  the  associated  program  for  acquiring  each  alternative. 

CRITICAL  ISSUES — The  aspects  of  a  system's  capability,  either  operational,  technical  or 
other,  that  must  be  questioned  before  a  system's  overall  worth  can  be  estimated,  and  are 
of  primary  importance  to  the  decision  authority  in  reaching  a  decision  to  allow  the  system 
to  advance  into  the  next  acquisition  phase. 

DATA  SYSTEM  —  Combinations  of  personnel  efforts,  forms,  formats,  instructions, 
procedures,  data  elements  and  related  data  codes,  communications  facilities  and  automatic 
data  processing  equipment  that  provide  an  organized  and  interconnected  means,  either 
automated,  manual  or  a  mixture  of  these  for  recording,  collecting,  processing  and  commu¬ 
nicating  data. 

DEFENSE  ACQUISITION  EXECUTIVE  (DAE)  —  The  principal  adviser  to  the  Secretary 
of  Defense  on  all  matters  pertaining  to  the  Department  of  Defense  Acquisition  Systems. 
The  Under  Secretary  of  Defense  for  Acquisition  is  the  DAE  and  the  Defense  Procurement 
Executive. 

CONCEPT  DEMONSTRATION  APPROVAL  —  Milestone  I  decision  by  which  the 
SECDEF  reaffirms  the  mission  need  and  approves  one  or  more  selected  alternatives  for 
competitive  demonstration  and  validation. 

DEPARTMENT  OF  DEFENSE  ACQUISITION  SYSTEM  —  A  single,  uniform  system 
whereby  all  equipment,  facilities  and  services  are  planned,  designed,  developed,  acquired, 
maintained  and  disposed  of  within  the  Department  of  Defense.  The  system  entails 
establishing  policies  and  practices  that  govern  acquisitions,  determining  and  prioritizing 
resource  requirements,  directing  and  controlling  the  process,  contracting  and  reporting  to 
the  Congress. 
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DESIGNATED  ACQUISITION  PROGRAM  —  Program  designated  by  the  Defense 
Acquisition  Executive  for  Defense  Acquisition  Board  milestone  review. 

DEVELOPER  EVALUATION  —  The  developer's  evaluation  addresses  all  aspects  of  the 
system  to  include  technical  performance,  operational  effectiveness  and  operational  suit¬ 
ability  of  cost  and  schedule. 

DEVELOPING  AGENCY  (DA)  —  The  Systems  Command  or  Chief  of  Naval  Operations 
designated  project  manager  assigned  responsibility  for  the  development,  test  and  evalua¬ 
tion  of  a  weapon  system,  subsystem  or  item  of  equipment. 

DEVELOPMENT  TEST  (DT)  —  A  technical  test  conducted  to  provide  data  on  safety,  the 
achievability  of  critical  system  technical  characteristics,  refinement  and  ruggedization  of 
hardware  configurations  and  determination  of  technical  risks.  This  testing  is  performed  on 
components,  subsystems,  materiel  improvement,  nondevelopment  items,  hardware-soft¬ 
ware  integration  and  related  software.  The  development  test  includes  the  testing  of 
compatibility  and  interoperability  with  existing  or  planned  equipment  and  systems  and 
the  system  effects  caused  by  natural  and  induced  environmental  conditions  during  the 
development  phases  of  the  materiel  acquisition  process. 

EARLY  OPERATIONAL  ASSESSMENT  —  An  operational  assessment  conducted  prior 
to,  or  in  support  of.  Milestone  II. 

EFFECTIVENESS — The  performance  or  output  received  from  an  approach  or  a  program. 
Ideally,  it  is  a  quantitative  measure  which  can  be  used  to  evaluate  the  level  of  performance 
in  relation  to  some  standard,  set  of  criteria  or  end  objective. 

ENGINEERING  CHANGE — An  alteration  in  the  physical  or  functional  characteristics  of 
a  system  or  item  delivered,  to  be  delivered  or  under  development,  after  establishment  of 
such  characteristics. 

ENGINEERING  CHANGE  PROPOSAL  (ECP)  —  Proposal  to  change  design  or  engineer¬ 
ing  features  of  materiel  under  development  or  production.  Includes  proposed  engineering 
change  and  documentation  by  which  the  change  is  described  and  suggested. 

ENGINEERING  DEVELOPMENT — The  RDTE  funding  category  that  includes  develop¬ 
ment  programs  being  engineered  for  Service  use  but  not  yet  approved  for  procurement  or 
operation.  Budget  Category  6.4  includes  those  projects  in  full-scale  development  of  Service 
use;  but  they  have  not  yet  received  approval  for  production  or  had  production  funds 
included  in  the  DOD  budget  submission  for  the  budget  or  subsequent  fiscal  year. 

EVALUATION  CRITERIA — Standards  by  which  achievement  of  required  technical  and 
operational  effectiveness/suitability  characteristics  or  resolution  of  technical  or  opera¬ 
tional  issues  maybe  evaluated.  Evaluation  criteria  should  include  quantitative  thresholds 
for  the  initial  operating  capability  (IOC)  system.  If  parameter  maturity  grows  beyond  KXI, 
intermediate  evaluation  criteria,  appropriately  time-lined,  must  also  be  provided. 
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FUTURE- YEAR  DEFENSE  PROGRAM  (FYDP)  —  The  official  document  which  summa¬ 
rizes  the  Secretary  of  Defense  approved  plans  and  programs  for  the  Department  of  Defense. 
It  is  published  at  least  annually. 

FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION  (FOT&E)  —  The  test  and 
evaluation  that  is  necessary  during  and  after  the  production  period  to  refine  estimates 
made  during  operational  test  and  evaluation,  to  evaluate  changes  and  to  reevaluate  the 
system  to  ensure  it  continues  to  meet  operational  needs  and  retains  its  effectiveness  in  a 
new  environment  or  against  a  new  threat. 

FOLLOW-ON  PRODUCTION  TEST  —  A  technical  test  conducted  subsequent  to  a  full 
production  decision  on  initial  production  and  mass  production  models  to  determine 
production  conformance  for  quality  assurance  purposes.  Program  funding  category  — 
Procurement. 

INDEPENDENT  EVALUATION  REPORT  —  A  report  that  provides  an  assessment  of 
item  or  system  operational  effectiveness  and  operational  suitability  vs.  critical  issues  as 
well  as  the  adequacy  of  testing  to  that  point  in  the  development  of  item  or  system. 

INDEPENDENT  OPERATIONAL  TEST  AGENCY  —  The  Army  Operational  Test  and 
Evaluation  Agency,  the  Navy  Operational  Test  and  Evaluation  Force,  the  Air  Force 
Operational  Test  and  Evaluation  Center,  and  the  Marine  Corps  Operational  Test  and 
Evaluation  Agency. 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (lOT&E)  —  All  operational  test 
and  evaluation  conducted  on  production  or  production-representative  articles  to  support 
the  decision  to  proceed  beyond  low-rate  initial  production.  It  is  conducted  to  provide  a 
valid  estimate  of  expected  system  operational  effectiveness  and  operational  suitability. 

IN-PROCESS  REVIEW  —  Review  of  a  project  or  program  at  critical  points  to  evaluate 
status  and  make  recommendations  to  the  decision  authority. 

INTEGRATED  LOGISTIC  SUPPORT  (ILS)  —  A  disciplined,  unified  and  iterative 
approach  to  the  management  and  technical  activities  necessary  to:  (a)  integrate  support 
considerations  into  system  and  equipment  design;  (b)  develop  support  requirements  that 
are  related  consistently  to  readiness  objectives,  design  and  each  other;  (c)  acquire  the 
required  support;  and  (d)  provide  the  required  support  during  the  operational  phase  at 
minimum  cost. 

INTEROPERABILITY — The  ability  of  systems,  units  or  forces  to  provide  services  to,  and 
accept  from,  other  systems,  units  or  forces,  and  to  use  the  services  so  exchanged  to  enable 
them  to  operate  together  effectively. 

ISSUES — Any  aspect  of  the  system's  capability,  either  operational,  technical  or  other,  that 
must  be  questioned  before  the  system's  overall  military  utility  can  be  known.  Operational 
issues  are  issues  that  must  be  evaluated  considering  the  soldier  and  the  machine  as  an  entity 
to  estimate  the  operational  effectiveness  and  operational  suitability  of  the  system  in  its 
complete  user  environment. 
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JOINT  DEVELOPMENT  TESTS  (JDTs) — The  JDTs  provide  information  on  intra-Service 
systems  or  equipment  requirements,  performance  or  interoperability;  technical  concepts, 
requirements  or  improvements;  and  the  improvement  or  development  of  testing  method¬ 
ologies  or  resources. 

JOINT  OPERATIONAL  TESTS  (JOTs)  —  The  JOTs  use  actual  fielded  equipment, 
simulators  or  surrogate  equipment  in  an  exercise  or  operational  environment  to  obtain  data 
pertinent  to  inter-Service  operational  doctrine,  tactics  and  procedures. 

LETHALITY  —  The  probability  that  weapon  effects  will  destroy  the  target  or  render  it 
neutral. 

LIFE-CYCLE  COST  —  The  total  cost  to  the  government  for  the  development,  acquisition, 
operation  and  logistic  support  of  a  system  or  set  of  forces  over  a  defined  life  span. 

LOGISTICS  SUPPORT  ABILITY  —  The  degree  of  which  the  planned  logistics  support 
(including  test  equipment,  measurement  and  diagnostic  equipment,  spare  and  repair 
parts,  technical  data,  support  facilities,  transportation  requirements,  training  and  man¬ 
power)  allow  meeting  system  availability  and  wartime  usage  requirements. 

LONG  LEAD  ITEMS  —  The  components  of  a  system  or  piece  of  equipment  that  take  the 
longest  time  to  procure  and,  therefore,  may  require  an  early  commitment  of  funds  in  order 
to  meet  acquisition  program  schedules. 

LOW  RATE  INITIAL  PRODUCTION  (LRIP)  —  Any  manufacture  of  a  system  in  limited 
quantity  to  be  used  in  initial  operational  test  and  evaluation  for  verifying  production 
engineering  and  design  maturity  and  to  establish  a  production  base. 

MAINTAINABILITY — A  characteristic  of  design  and  installation  that  N  expressed  as  the 
probability  of  an  item  being  retained  in,  or  restored  to,  a  specified  condition  within  a  given 
period  of  time  when  the  maintenance  is  performed  in  accordance  with  prescribed  proce¬ 
dures  and  resources. 

MAJOR  DEFENSE  ACQUISITION  PROGRAM — As  specified  in  United  States  Code  10, 
sections  136a  and  139a  (reference  1)  and  DOD  Directive  5000.1: 

a.  A  DOD  acquisition  program  that  is  not  a  highly  sensitive/classified  program  (as 
determined  by  the  Secretary  of  Defense)  and: 

(1)  That  is  designated  by  the  Secretary  of  Defense  as  a  major  defense  acquisition 
program;  or 

(2)  That  is  estimated  by  the  Secretary  of  Defense  to  require  an  eventual  total 
expenditure  for  research,  development,  test  and  evaluation  of  more  than  200 
million  dollars  (based  on  fiscal  year  1980  constant  dollars)  or  an  eventual  total 
expenditure  for  procurement  of  more  than  1  billion  dollars  (based  on  fiscal  year 
1980  constant  dollars). 
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b.  A  DOD  acquisition  program  that  is  designated  jointly  by  the  DOT&E  and  DTE, 
as  a  major  defense  acquisition  program  for  the  purpose  of  carrying  out  the 
responsibilities,  fimctions,  and  authorities  of  this  Manual.  Such  designation  for 
the  purpose  of  Test  and  Evaluation  oversight  does  not  imply  any  other  related 
review  requirements. 

MAJOR  RANGE  AND  TEST  FACILITY  BASE  (MRTFB)  —  The  complex  of  major  DOD 
ranges  and  test  facilities. 

MILESTONE  —  A  major  management  decision  point  in  the  overall  acquisition  process  of 
a  major  Department  of  Defense  (DOD)  system  requiring  Office  of  the  Secretary  of  Defense 
and/or  DOD  Component  Program  review.  Milestones  include  Joint  Resource  Manage¬ 
ment  Board  and  DOD  Component  Equivalent  Program  reviews. 

MILITARY  REQUIREMENT  —  An  established  need  justifying  the  timely  allocation  of 
resources  to  achieve  a  capability  to  accomplish  approved  military  objectives,  missions  or 
tasks.  Requirements  are  normally  documented  in  a  Mission  Needs  Statement  or  Opera¬ 
tional  Requirement  Document. 

MISSION  AREA  ANALYSIS  (MAA) — Continuous  analysis  of  assigned  mission  respon¬ 
sibilities  in  the  several  mission  areas  to  identify  deficiencies  in  the  current  and  projected 
capabilities  to  meet  essential  mission  needs  and  to  identify  opportunities  for  the  enhance¬ 
ment  of  capability  through  more  effective  systems  and  less-costly  methods. 

MISSION  NEED  STATEMENT  (MNS) — Submitted  prior  to  Program  Objectives  Memo¬ 
randum  submission.  Approval  by  SECDEF  is  Milestone  0.  Documents  major  mission 
deficiencies  (or  opportunities  for  improvement)  in  a  Service's  ability  to  meet  mission 
requirements  when  such  deficiencies  can  be  corrected  by:  (1)  using  an  existing  U.S.  system 
or  allied  military  or  commercial  system,  (2)  a  major  modification  to  an  existing  system,  or 
(3)  a  new  major  acquisition.  A  joint  MNS  is  prepared  to  document  major  deficiencies  in  two 
or  more  DOD  components.  The  Office  of  the  Secretary  of  Defense  or  Office  of  the  Joint 
Chiefs  of  Staff  may  also  prepare  the  MNS. 

MISSION  RELIABILITY — The  probability  that  the  system  will  perform  mission  essential 
functions  for  a  period  of  time  under  the  conditions  stated  in  the  mission  profile. 

MODEL  —  A  model  is  a  representation  of  an  actual  or  conceptual  system  involving 
mathematics,  logical  expressions  or  computer  simulations  that  can  be  used  to  predict  how 
the  system  might  perform  or  survive  under  various  conditions  or  in  a  range  of  hostile 
environments. 

MULTI-SERVICE  OPERATIONAL  TEST  —  A  form  of  test  when  one  or  more  of  the 
Services  provide  support  Service  test  or  vice  versa  or  tests  that  involve  agreements  between 
a  Service  and  one  or  more  of  the  other  Services  to  evaluate  a  system  or  concept  that  requires 
testing  in  a  multi-Service  environment. 

NONDEVELOPMENT  ITEM  (NDI)  —  Already  developed  and  available  hardware  and/ 
or  software  capable  of  fulfilling  Service  requirements,  thereby  minimizing  or  eliminating 
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the  need  for  costly,  time-consuming,  government-sponsored  R&D  programs.  An  NDI  is 
usually  off-the-shelf  or  a  commercial-type  product,  but  may  also  be  equipment  already 
developed  by  or  for  the  military  services  or  foreign  military  forces. 

NUCLEAR  HARDNESS  —  A  quantitative  description  of  the  physical  attributes  of  the 
system  or  component  that  will  allow  nuclear  survivability  in  a  given  weapon  environment. 
Hardness  is  measured  by  physical  quantities  such  as  overpressure,  peak  velocities,  energy 
absorbed,  electrical  stress,  etc.  Hardness  is  achieved  through  design  specifications  and 
often  verified  by  one  or  more  test  and  analysis  techniques.  Hardness  is  only  one  of  several 
means  of  attaining  system-wide  nuclear  survivability. 

OPERATIONAL  ASSESSMENT — An  evaluation  of  operational  effectiveness  and  opera¬ 
tional  suitability  made  by  an  independent  operational  test  activity,  with  user  support  as 
required,  on  other  than  production  systems.  The  focus  of  an  operational  assessment  is  on 
significant  trends  noted  in  development  efforts,  programmatic  voids,  areas  of  risk,  ad¬ 
equacy  of  requirements  and  the  ability  of  the  program  to  support  adequate  operational 
testing.  Operational  assessments  may  be  made  at  any  time  using  technology  demonstra¬ 
tors,  prototypes,  mock-ups,  engineering  development  models  or  simulations  but  will  not 
substitute  for  the  independent  operational  test  and  evaluation  necessary  to  support  full 
production  decisions. 

OPERATIONAL  AVAILABILITY  —  An  index  of  weapon  system  materiel  readiness, 
including  system  software  where  applicable,  in  a  mission  environment.  It  is  a  measure  of 
probability  of  an  item's  being  in  a  condition,  generally  referred  to  as  "up,"  so  it  can  perform 
its  intended  function  when  called  upon  within  acceptable  limits  of  degradation. 

OPERATIONAL  EFFECTIVENESS  —  The  overall  degree  of  mission  accomplishment  of 
a  system  when  used  by  representative  personnel  in  the  environment  planned  or  expected 
(e.g.  natural,  electronic,  threat,  etc.)  for  operational  employment  of  the  system  considering 
organization,  doctrine,  tactics,  survivability,  vulnerability  and  threat  (including  counter¬ 
measures;  initial  nuclear  weapons  effects;  and  nuclear,  biological  and  chemical  contamina¬ 
tion  threats). 

OPERATIONAL  EVALUATION  —  Addresses  the  effectiveness  and  suitability  of  the 
weapons,  equipment  or  munitions  for  use  in  combat  by  typical  military  users  and  the 
system  operational  issues  and  criteria;  provides  information  to  estimate  organizational 
structure,  personnel  requirements,  doctrine,  training  and  tactics;  identifies  any  operational 
deficiencies  and  the  need  for  any  modifications;  and  assesses  MANPRINT  (safety,  health 
hazards,  human  factors,  manpower  and  personnel)  aspects  of  the  system  in  a  realistic 
operational  environment. 

OPERATIONAL  SUITABILITY  —  The  degree  to  which  a  system  can  be  placed  satisfac¬ 
torily  in  field  use,  with  consideration  being  given  to  availability,  compatibility,  transport¬ 
ability,  interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportability,  logistic  supportability  and  training  requirements. 

OPERATIONAL  TEST — Testing  of  materiel  systems  that  is  accomplished  with  represen¬ 
tative  user  operators,  crews,  support  personnel  or  units  in  as  realistic  an  operational 
environment  as  possible  to  provide  the  evaluator  data  to  estimate: 
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a.  The  military  operational  effectiveness  and  operational  suitability  (including  com¬ 
patibility,  interoperability,  reliability,  availability,  maintainability,  supportability, 
operational  soldier/hardware/software  interface  and  training  requirements)  of  new 
systems. 

b.  The  system's  desirability,  from  the  use  viewpoint,  considering  systems  already 
available  and  the  operational  benefits  and/or  burdens  associated  with  the  new 
system. 

c.  The  need  for  modifying  the  system. 

d.  The  adequacy  of  doctrine,  organization,  operating  techniques,  tactics  and  training 
for  employment  of  the  system;  the  adequacy  of  maintenance  and  supply  support  for 
the  system;  and,  when  appropriate,  its  performance  in  a  countermeasure  environ¬ 
ment. 

OPERATIONAL  TEST  AND  EVALUATION  (OT&E)  —  The  field  test,  under  realistic 
combat  conditions,  of  any  item  (or  key  component  of),  weapons,  equipment  or  munitions 
for  the  purpose  of  determining  the  effectiveness  and  suitability  of  the  weapons,  equipment 
or  munitions  for  use  in  combat  by  typical  military  users;  and  the  evaluation  of  the  results 
of  such  test. 

OPERATIONAL  TEST  CRITERIA — Expressions  of  the  operational  level  of  performance 
required  of  the  military  system  to  demonstrate  operational  effectiveness  for  given  func¬ 
tions  during  each  operational  test.  The  expression  consists  of  the  function  addressed,  the 
basis  for  comparison,  the  performance  required  and  the  confidence  level. 

OPERATIONAL  TEST  READINESS  REVIEW  (OTRR)  —  A  review  to  identify  problems 
that  may  impact  the  conduct  of  an  OT&E.  The  OTRRs  are  conducted  to  determine  changes 
required  in  planning,  resources  or  testing  necessary  to  proceed  with  the  OT&E.  Participants 
include  the  operational  tester  (chair),  evaluator,  material  developer,  user  representative, 
logisticians,  HQDA  staff  elements  and  others,  as  necessary. 

PILOT  PRODUCTION  —  The  controlled  manufacture  of  limited  numbers  of  an  item  for 
Service  test  and  evaluation  purposes  using  manufacturing  drawings  and  specifications, 
which  have  been  developed  for  quantity  production  and  with  tooling  that  is  representative 
of  that  to  be  used  in  unlimited  production. 

POSTPRODUCTION  TESTING  —  Testing  conducted  to  ensure  that  materiel  that  is 
reworked,  repaired,  renovated,  rebuilt  or  overhauled  after  initial  issue  and  deployment 
conforms  to  specified  quality,  reliability,  safety  and  operational  performance  standards. 
Included  in  postproduction  tests  are  surveillance  tests,  stockpile  reliability  and  recondi¬ 
tioning  tests. 

PREPLANNED  PRODUCT  IMPROVEMENT  (P^I)  —  Planned  future  evolutionary 
improvement  of  developmental  systems  for  which  design  considerations  are  effected 
during  development  to  enhance  future  application  of  projected  technology.  Includes 
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ments  planned  for  ongoing  systems  that  go  beyond  the  current  performance  envelope  to 
achieve  a  needed  operational  capability. 

PREPRODUCTION  PROTOTYPE  —  Items  in  final  form  employing  standard  parts  that 
are  representative  of  items  to  be  produced  on  a  production  line  with  production  tooling. 

PRODUCT  IMPROVEMENT  PLAN  (PIP)  —  Effort  to  incorporate  configuration  changes 
involving  engineering  and  testing  effort,  on  end-items  and  depot  repairable  components 
or  changes  on  other-than-developmental  items  to  increase  system  or  combat  effectiveness 
or  extend  the  useful  military  life. 

PRODUCTION  QUALIFICATION  TEST  (PQT)  —  A  technical  test  conducted  post- 
Milestone  III  to  ensure  the  effectiveness  of  the  manufacturing  process,  equipment  and 
procedures.  This  testing  also  provides  data  for  the  independent  evaluation  required  for 
materiel  release  so  the  evaluator  can  address  the  adequacy  of  the  materiel  with  respect  to 
the  stated  requirements.  These  tests  are  conducted  on  a  number  of  samples  taken  at  random 
from  the  first  production  lot  and  are  repeated  if  the  process  or  design  is  changed 
significantly  and  when  a  second  or  alternative  source  is  brought  on  line  Program  funding 
category  —  Procurement. 

PROGRAM  MANAGER — Individual  chartered  by  the  Service  Secretary  reporting  to  the 
material  developer  or  to  the  commander  of  a  subordinate  organization  as  designated  by  the 
material  developer.  Assigned  responsibility  and  delegated  full-line  authority  of  the  mate¬ 
rial  developer  for  centralized  management  of  a  specified  acquisition  or  materiel  readiness 
program.  May  be  superimposed  over  one  or  more  product  managers. 

QUALIFICATIONS  TESTING  —  Testing  that  verifies  the  design  and  manufacturing 
process  and  provides  a  baseline  for  subsequent  acceptance  tests.  The  completion  of 
production  qualification  test  and  evaluation  before  Milestone  III  decisions  is  essential  and 
will  be  a  critical  factor  in  assessing  the  system's  readiness  for  production.  Production 
qualification  test  and  evaluation  shall  be  conducted  on  production-representative  items. 

QUALITY  ASSURANCE  —  A  planned  and  systematic  pattern  of  all  actions  necessary  to 
provide  adequate  confidence  that  materiel  conforms  to  established  technical  requirements 
and  achieves  satisfactory  performance  in  service. 

REALISTIC  TEST  ENVIRONMENT — The  conditions  under  which  a  system  is  expected 
to  be  operated  and  maintained,  including  the  natural  weather  and  climatic  conditions, 
terrain  effects,  battlefield  disturbances  and  enemy  threat  conditions. 

RELIABILITY  —  The  probability  that  an  item  will  perform  its  intended  function  for  a 
specified  interval  under  stated  conditions. 

REQUIRED  OPERATIONAL  CHARACTERISTICS  —  Qualitative  and  quantitative 
system  parameters  approved  by  the  user  that  are  primary  indicators  of  a  system's 
capability  to  accomplish  its  mission  (operational  effectiveness)  and  to  be  supported 
(operational  suitability). 
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REQUIRED  TECHNICAL  CHARACTERISTICS  —  Quantitative  system  parameters 
approved  by  the  Department  of  Defense  Component  that  are  selected  as  primary  in¬ 
dicators  of  technical  achievement  of  engineering  thresholds.  These  might  not  be  direct 
measures  of,  but  should  always  relate  to,  a  system's  capability  to  perform  its  required 
mission  function  and  to  be  supported 

RESEARCH  —  Includes  all  effort  of  scientific  study  and  experimentation  directed  toward 
increasing  knowledge  and  understanding  in  the  physical,  engineering,  environmental  and 
life  sciences  fields  related  to  long-term  national  security  needs.  It  provides  fundamental 
knowledge  required  for  the  solution  of  military  problems.  It  forms  a  part  of  the  base  for 
subsequent  exploratory  and  advanced  developments  in  defense-related  technologies,  and 
new  and  improved  military  functional  capabilities  in  areas  such  as  communications, 
detection,  tracking,  surveillance,  propulsion,  mobility,  guidance  and  control,  navigation, 
energy  conversion,  materials  and  structures  and  personnel  support.  (Budget  Category  6.1) 

RISK  —  An  expression  of  possible  loss  in  terms  of  hazard  severity  and  hazard  probability. 

RISK  ASSESSMENT  —  An  evaluation  of  a  risk  in  terms  of  mission  loss  should  a  hazard 
result  in  an  accident. 

SAFETY  —  Freedom  from  those  conditions  that  can  cause  death,  injury,  occupational 
illness  or  cause  damage  to,  or  loss  of,  equipment  or  property. 

SAFETY/HEALTH  VERIFICATION  —  The  development  of  data  used  to  evaluate  the 
safety  and  health  features  of  a  system  to  determine  its  acceptability.  This  is  done  primarily 
during  developmental  test  and  user  or  operational  test  and  evaluation  and  supplemented 
by  analysis  and  independent  evaluations. 

SAFETY  RELEASE  —  A  formal  document  issued  to  a  user  test  organization  before  any 
hands-on  use  or  maintenance  by  personnel.  The  safety  release  indicates  the  system  is  safe 
for  use  and  maintenance  by  typical  user  personnel  and  describes  the  system  safety 
analyses.  Operational  limits  and  precautions  are  included.  The  test  agency  uses  the  data  to 
integrate  safety  into  test  controls  and  procedures  and  to  determine  if  the  test  objectives  can 
be  met  within  these  limits. 

SELECTED  ACQUISITION  REPORT  —  A  standard,  comprehensive,  summary  status 
report  provided  to  the  Congress  on  Department  of  Defense  ( DOD)  acquisition  programs 
for  management  within  DOD. 

SIMULATION  —  A  simulation  is  a  method  for  implementing  a  model.  It  is  the  process  of 
conducting  experiments  with  a  model  for  the  purpose  of  understanding  the  behavior  of  the 
system  modeled  under  selected  conditions  or  limits  imposed  by  developmental  or  opera¬ 
tional  criteria.  Simulation  may  include  the  use  of  analog  or  digital  devices,  laboratory 
models  or  "test-bed"  sites.  Simulations  are  usually  programmed  for  solution  on  a  com¬ 
puter;  however,  in  the  broadest  sense,  military  exercises  and  war  games  are  also  simula¬ 
tions. 
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SPECIFICATION  —  A  specific  quantitative,  contractually  binding,  required  operational 
or  technical  characteristic. 

SUBTEST  —  An  element  of  a  test  program.  A  subset  is  a  test  conducted  for  a  specific 
purpose  (e.g.,  rain,  dust,  transportability,  missile  firing,  fording). 

SUIT  ABILITY — A  subjective  determination  by  a  decision  authority  that  a  materiel  system 
does  or  does  not  meet  minimum  standards  prerequisite  to  satisfy  field  Service  use.  The 
judgment  may  be  based  on  the  presence  or  absence  of  uncorrectable  materiel  deficiencies 
and/or  the  number  and  assessed  importance  of  correctable  and  uncorrectable  shortcom¬ 
ings.  It  also  includes  judgments  on  nonmateriel  issues. 

SURVIVABILITY  —  The  capability  of  a  system  to  avoid  or  withstand  man-made  hostile 
environments  without  suffering  an  abortive  impairment  of  its  ability  to  accomplish  its 
designated  mission. 

SUSCEPTIBILITY — The  degree  to  which  a  device,  equipment  or  weapon  system  is  open 
to  effective  attack  due  to  one  or  more  inherent  weaknesses.  (Susceptibility  is  a  function  of 
operational  tactics,  coimtermeasures,  probability  of  enemy  fielding  a  threat,  etc.) 

SYSTEM  —  A  composite,  at  any  level  of  complexity,  of  personnel,  procedures,  materials, 
tools,  equipment,  facilities  and  software.  The  elements  of  this  composite  entity  are  used 
together  in  the  intended  operational  or  support  environment  to  perform  a  given  task  or 
achieve  a  specific  production,  support  or  mission  requirement. 

SYSTEM  ENGINEERING,  DEFENSE  —  That  portion  of  the  acquisition  process  dealing 
with  the  transformation  of  an  operational  need  into  an  optimal  set  of  system  performance 
pareuneters  and  a  preferred  system  configuration.  It  includes  engineering/ technical  man¬ 
agement,  definition  of  system  and  program,  design  engineering,  support  engineering,  the 
integration  of  the  engineering  specialties,  and  o3\er  such  factors  that  affect  the  develop¬ 
ment,  production,  deployment,  operation  and  disposal  of  the  system. 

SYSTEM  ENGINEERING  PROCESS  —  A  logical  sequence  of  activities  and  decisions 
transforming  an  operational  need  into  a  description  of  system  performance  parameters  and 
a  preferred  system  configuration. 

TECHNICAL  EVALUATION  —  Addresses  the  system's  technical  issues  and  criteria  and 
the  acquisition  and  fielding  of  an  effective,  supportable  and  safe  system.  It  assists  in  the 
engineering  design  and  development  and  verifies  attainment  of  technical  performance 
specifications,  objectives,  producibility,  adequacy  of  the  Technical  Data  Package,  and 
supportability,  determining  safety,  health  hazards,  human  factors,  and  MAl^RINT 
aspects.  Technical  evaluation  encompasses  the  use  of  models,  simulations  and  test  beds  as 
well  as  a  prototypes  or  full-scale  development  models  of  the  system. 

TECHNICAL  FEASIBILITY  TEST — A  technical  test  conducted  post-Milestone  0  and  pre- 
Milestone  I  or  Milestone  I/II  (under  the  Army  Streamlined  Acquisition  Process)  to  cissist 
in  determining  safety  and  establishing  system  performance  specifications  and  feasibility. 
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TECHNICAL  TESTER  —  The  command  or  agency  that  plans,  conducts  and  reports  the 
results  of  Army  technical  testing.  Associated  contractors  may  perform  development  testing 
on  behalf  of  the  command  or  agency. 

TECHNICAL  TESTS — A  generic  Army  term  for  testing  that  gathers  technical  data  during 
development  testing,  technical  feasibility  testing,  qualification  testing,  joint  development 
testing  and  contractor/foreign  testing.  Soldier  operator-maintainer  test  and  evaluation 
personnel  are  used  during  technical  testing  when  appropriate. 

TEST  AND  EVALUATION  MASTER  PLAN  —  An  overall  test  and  evaluation  strategy 
plan  that  is  prepared  as  early  as  possible  in  the  acquisition  process  and  is  designed  to 
identify  and  integrate  objectives,  responsibilities,  resources  and  schedule  for  all  test  and 
evaluation  to  be  accomplished  prior  to  key  decision  milestones. 

TEST  BEDS  —  A  system  representation  consisting  partially  of  actual  hardware  and 
partially  of  computer  models  or  prototype  hardware. 

TEST  CRITERIA  —  Standards  by  which  test  results  and  outcome  are  judged. 

TEST  DESIGN  PLAN  —  A  statement  of  the  conditions  under  which  t  j  test  is  to  be 
conducted,  the  data  required  from  the  test,  and  the  data  handling  required  to  relate  the  data 
results  to  the  test  conditions. 

TEST  INSTRUMENTATION  —  Test  instrumentation  is  e-rientific,  automated  data  pro¬ 
cessing  equipment  or  technical  equipment  used  to  measure,  sense,  record,  transmit, 
process  or  display  data  during  tests,  evaluations  or  examination  of  materiel,  training 
concepts,  or  tactical  doctrine.  Audiovisual  is  included  as  instrumentation  when  used  to 
support  Army  testing. 

TEST  RESOURCES  —  A  collective  term  that  encompasses  all  elements  necessary  to  plan, 
conduct  and  collect/ analyze  data  from  a  test  event  or  program.  Elements  include  test 
funding  and  support  manpower  (including  TDY  costs),  test  assets  (or  units  under  test),  test 
asset  support  equipment,  technical  data,  simulation  models,  test  beds,  threat  simulators, 
surrogates  and  replicas,  special  instrumentation  peculiar  to  a  given  test  asset  or  test  event, 
targets,  tracking  and  data  acquisition,  instrumentation,  equipment  for  data  reduction, 
communications,  meteorology,  utilities,  photography,  calibration,  security,  recovery, 
maintenance  and  repair,  frequency  management  and  control,  and  base/facility  support 
services. 

THREAT — The  sum  of  the  potential  strength,  capabilities  and  intentions  of  an  enemy  that 
can  limit  or  negate  mission  accomplishment  or  reduce  force,  system  or  equipment  effective¬ 
ness. 

THRESHOLDS  —  The  minimum  level  of  a  performance  parameter  the  system  must  meet 
(e.g.,  minimum  flight  altitude  threshold  of  30  feet  above  ground  level  for  a  missile). 
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TRANSPORTABILITY — The  inherent  capability  of  materiel  to  be  moved  by  towing,  self¬ 
propulsion  or  carrier  via  railways,  highways,  waterways,  pipelines,  ocean  and  airways. 

USER  REPRESENTATIVE  —  The  combat  developer  designated  to  represent  the  user 
during  the  materiel  acquisition  process.  The  command  or  agency  fulfilling  this  role 
represents  the  "mission-oriented"  user  and  the  "logistics-oriented"  user  by  concerning 
itself  with  the  operational  and  logistic  support  aspects  of  materiel  system. 

VULNERABILITY  —  The  characteristics  of  a  system  that  cause  it  to  suffer  a  definite 
degradation  (loss  or  reduction  of  capability  to  perform  the  designated  mission)  as  a  result 
of  having  been  subjected  to  a  certain  (defined)  level  of  effects  in  an  unnatural  (man-made), 
hostile  environment. 

WORK  BREAKDOWN  STRUCTURE  (WBS)  —  A  product-oriented  family-tree  division 
of  hardware,  software,  services  and  other  work  tasl^  that  organizes,  defines  and  graphi¬ 
cally  displays  the  product  to  be  produced  as  well  as  the  work  to  be  accomplished  to  achieve 
the  specified  product. 
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APPENDIX  C 

TEST-RELATED  DATA  ITEM  DESCRIPTIONS 

extracted  from  DOD  5010.12-L, 

Acquisition  Management  System  and 
Data  Requirement  Control  List  (AMSDL) 


ACCEPTANCE  TEST  PLAN 

AIRBORNE  SOUND  MEASUREMENTS  TEST  REPORT 

AIRFRAME  RIGIDITY  TEST  REPORT 

AMMUNITION  TEST  EXPENDITURE  REPORT 

ARMOR  MATERIAL  TEST  REPORTS 

BALLISTIC  ACCEPTANCE  TEST  REPORT 

C.P.  PROPELLER  TEST  AGENDA 

COORDINATED  TEST  PLAN 

CORROSION  TESTING  REPORTS 

DAMAGE  TOLERANCE  TEST  RESULTS  REPORTS 

DEMONSTRATION  TESTrPLAN 

REPORT 

DIRECTED  ENERGY  SURVIVABILITY  TEST  PLAN 

DURABILITY  TEST  RESULTS  REPORT 

ELECTROMAGNETIC  COMPATIBILITY  TEST  PLAN 

ELECTROMAGNETIC  INTERFERENCE  TEST:PLAN 

REPORT 

ELECTROSTATIC  DISCHARGE  SENSITIVITY  TEST  REPORT 
EMISSION  CONTROL  (EMCON)  TEST  REPORT 
ENDURANCE  TEST  (EMCS)  FAILURE  REPORTS 


DI-QCIC-80154, 

-80553 

DI-HFAC-80272 

DI-T-30734 

DI-MISC-80060 

DI-MISC-80073 

DI-MISC-80246 

UDI-T-23737 

DI-MGMT-80937 

DI-MFFP-80108 

DI-T.30725 

DI-QCIC-80775 

DI-QCIC-80774 

DI-R-1786 

DI-T-30726 

DI-T-3704B 

DI-EMCS-80201 

DI-EMCS-80200 

DI-RELI-80670 

DI-R-2059 

DI-ATTS-80366 
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ENGINEER  DESIGN  TEST  PLAN 

DI-MGMT-80688 

ENVIRONMENTAL  DESIGN  TEST  PLAN 

DI-ENVR-80861 

ENVIRONMENTAL  TEST  REPORT 

EQUIPMENT  TEST  PLAN  (NONSYSTEM) 

DI-ENVR-80863 

DI-T-3709A 

FACTORY  TEST:  PLAN 

EMCSPLAN 

EMCS  PROCEDURES 

EMCS  REPORTS 

DI-QCIC-80153 

DI-ATrS-80360 

DI-ATTS-80361 

DI-ATTS-80362 

FIRST  ARTICLE  QUALIFICATION  TEST  PLAN 

DI-T-5315A 

FLIGHT  FLUTTER  TEST  REPORT 

DI-T-30733 

FLUTTER  MODEL  TEST  REPORT 

DI-T-30732 

HARDWARE  DIAGNOSTIC  TEST  SYSTEM 

DEVELOPMENT  PLAN 

DI-ATTS-80005 

HIGH-IMPACT  SHOCK  TEST  PROCEDURES 

DI-ENVR-80709 

HULL  TEST  RESULTS  (BOATS)  REPORT 

UDI-T-23718 

HUMAN  ENGINEERING  TEST:  PLAN 

REPORT 

DI-HFAC-80743 

DI-HFAC-80744 

INSPECTION  AND  TEST  PLAN 

DI-QCIC-81110 

INSTALLATION  TEST:  PLAN 

PROCEDURES 

REPORT 

DI-QCIC-80155 

DI-QCIC-80511 

DI-QCIC-80140, 

-80512 

INTEGRATED  CIRCUIT  TEST  DOCUMENTATION 

DI-ATTS-80888 

MAINTAINABILITY/TESTABILITY  DEMONSTRATION 

TEST:  PLAN 

REPORT 

DI-MNTY-80831 

DI-MNTY-80832 

MAINTENANCE  TRAINING  EQUIPMENT  TEST  OUTLINES 

DI-H-6129A 

MASTER  TEST  PLAN/PROGRAM  TEST  PLAN 

DI-T-30714 

NBC  CONTAMINATION  SURVIVABILITY  TEST:  PLAN 

DI-R-1779 
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NUCLEAR  SURVIVABILITY  TEST:  PLAN 

REPORT 

DI-NUOR-80928 

DI-NUOR-80929 

PACKAGING  TEST:  PLAN 

REPORT 

DI-PACK-80456 

DI-PACK-80457 

PART,  COMPONENT  OR  SUBSYSTEM  TEST  PLAN(S) 

DI-MISC-80759 

PARTS  (NONSTANDARD)  TEST  DATA  REPORT 

DI-MISC-81058 

PARTS  QUALIFICATION  TEST  PLAN 

DI-T-5477A 

PERFORMANCE  ORIENTED  PACKAGING  TEST  REPORT 

DI-PACK-81059 

PRODUCTION  TEST:  PLAN 

REPORT 

DI-MNTY-80173 

DI-NDTI-80492 

QUALITY  CONFORMANCE  TEST  PROCEDURES 

DI-RELI-80322 

RADAR  SPECTRUM  MANAGEMENT  (RSM)  TEST  PLAN 

DI-MISC-81113 

RANDOMIZER  TEST  REPORT 

DI-NDTI-80884 

RELIABILITY  TEST:  PLAN 

PROCEDURES 

REPORTS 

DI-RELI-80250 

DI-RELI-80251 

DI-RELI-80252 

RESEARCH  AND  DEVELOPMENT  TEST  AND  ACCEPTANCE 
PLAN 

DI-T-30744 

ROUGH  HANDLING  TEST  REPORT 

DI-T-5144C 

SHIP  ACCEPTANCE  TEST  (SAT):  SCHEDULE 

REPORT 

DI-T-23959B 

DI-T-23190A 

SHIPBOARD  INDUSTRIAL  TEST  PROCEDURES 

DI-QCIC-80206 

SHOCK  TEST:  EXTENSION  REQUEST 

REPORT 

DI-EN\^R-80706 

DI-ENVR-80708 

SOFTWARE  GENERAL  UNIT  TEST  PLAN 
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PLANS/PROCEDURES 

PROCEDURE 

PROCEDURES 

TEST  PLAN  DOCUMENTATION  FOR  AIS 

TEST  PROGRAM:  DOCUMENTATION  (TPD) 

INTEGRATION  LOGBOOK 

TPS  AND  OTPS  ACCEPTANCE  TEST:  PROCEDURES  (ATPS) 

REPORT  (ATR) 


TEST  REPORTS 


TEST  REQUIREMENTS  DOCUMENT 
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DI-MCCR-80308 

DI-HFAC-80271 
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DI-R-2068 

DI-T-21463A 

DI-T-21464A 

DI-HFAC-80274 


DI-T-5463A 

DI-EMCS-80218 

DI-T-1912A 

DI-T-26391B 

DI-QCIC-80204 

DI-FACR-80810 

DI-ILSS-81085 

DI-NDTI-80566 

DI-NDTI-80808 

DI-NDTI-80603 

UDI-T-23732B 

DI-IPSC-80697 

DI-ATTS-80284 
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TEST  SCHEDULING  REPORT 
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TESTABILITY:  PROGRAM  PLAN 

ANALYSIS  REPORT 
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TRAINER  TEST  PROCEDURES  AND  RESULTS  REPORT 

DI-T-25594C 

VIBRATION  AND  NOISE  TEST  REPORTS 

DI-T-30735 

VIBRATION  TESTING:  EXTENSION 

REPORT 

UDI-T-23752 

UDI-T-23762 

WELDING  PROCEDURE  QUALIFICATION  TEST  REPORT 

DI-MISC-80876 

STANDARDIZATION  AREAS 

LEAD  SERVICE 

ATTS 

Automatic  Test  Technology  Standards 

10 

CMAN 

Configuration  Management 

SD 

E 

Engineering  and  Configuration  Documentation 

EMCS 

Electromagnetic  Compatibility 

EC 

ENVR 

Environmental  Requirements  and  Related  Test  Meth 

TE 

FACR 

Facility  Construction  Design  Requirements 

YD 

GDRQ 

General  Design  Requirements 

SD 

H 

Human  Factors 

HFAC 

Human  Factors 

MI 

ILSS 

Integrated  Logistics  Support  Standards 

WS 

IPSC 

Information  Processing  Standards  for  Computers 

02 

MCCR 

Mission  Critical  Computer  Resources 

10 

MFFP 

Metal  Finishes  and  Finishing  Processes  and  Proc 

MR 

MGMT 

Management 

var 

MISC 

Miscellaneous 

SD 

MNTY 

Maintainability 

17 
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NDn 

Nondestructive  Testing  and  Inspection 

MR 

NUOR 

Nuclear  Ordnance 

DS 

PACK 

Packing,  Packaging,  Preservation  and  Transport 

SM 

QCIC 

Quality  Control/ Assurance  and  Inspection 

AR 

R 

Related  Design  Requirements 

RELI 

Reliability 

17 

S 

System/Subsystem  Analysis 

« 

STANDARDIZATION  AREAS 

LEAD  SERVICE 

SAFT 

Safety 

10 

TMSS 

Technical  Manual  Specifications  and  Standards  TM 

T 

Test 

V 

Test 

*Prior  to  1  Jul  1985;  being  attritted  out. 

UDI:  Indicates  DID  unique  to  originator;  being 
attritted  out. 


Lead  Services 


02 

USAF 

ACS  Information  Systems 

10 

USAF 

AFMC  Command  Standardization  Office 

17 

USAF 

AFMC  Rome  Air  Development  Center 

AR 

USA 

AMC  AMCCOM  ARDEC 

DS 

DNA 

Defense  Nuclear  Agency 

EC 

USN 

SPAWAR 

MI 

USA 

AMC  MICOM 

MR 

USA 

AMC  ARL  Matl  Tech  Lab 
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SD 

OSD 

Standardization  and  Data  Management 

SM 

USA 

AMC  Packaging,  Storage  and  Containerization  Center 

TE 

USA 

AMC  TECOM 

TM 

USA 

AMC  Matl  Readiness  Spt  Activity 

WS 

OSD 

Wpn  Spt  Improvement  and  Analysis  Office 

YD 

USN 

Nav  Fac  Engr  Cmd 
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Defense  (Live  Fire  Test)  Before  the  Acquisition  Policy  Panel  of  the  House 
Armed  Services  Committee,  September  10, 1987. 

35.  Memorandum  of  Agreement  of  Multi-Service  OT&E  and  Joint  T&E,  with 
changes. 

36.  Joint  Test  and  Evaluation  Procedures  Manual,  (Draft),  Office  of  the  Deputy 
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44.  Joint  Logistics  Commanders  Guide  for  the  Management  of  Joint  Service 
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71 .  (cancelled)  OPNAVINST  3960.10C,  Test  and  Evaluation  (Draft). 
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Evaluation  Force  (OPTEVFOR). 

74.  NAVMATINST  3960,  Test  and  Evaluation. 

75.  COMOPTEVFORINST  3960.1D,  Operational  Test  Director's  Guide. 
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78.  COMOPTEVFOR,  AD  Number  AD124168,  Operational  Test  Director  Guide. 
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82.  (cancelled)  AFR  80-14,  Test  and  Evaluation. 
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84.  AFR  800-2,  Acquisition  Program  Management. 

85.  AFR  800-8,  Integrated  Logistic  Support  (ILS)  Program. 

86.  CAFSCR  550-15,  Statistical  Process  Control. 

87.  AFSCR  550-13,  Producibility. 

88.  AFOTEC  Regulation  23-1,  Organization  and  Functions  Air  Force  Operational 
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90.  AFOTEC,  Electronic  Warfare  Operational  Test  and  Evaluation. 
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AIR  FORCE 


OSD 


Commander  U.S.  Army  Test  and  Evaluation  Command 
ATTN:  AMSTE-TE 

Aberdeen  Proving  Groimd,  MD  21005-5055 

Commander  U.S.  Army  Logistics  Management  Group 
ATTN:  AMXMC-ACM-MA 
Fort  Lee,  VA  23801-6048 

Commander  U.S.  Army  Operational  Test  &  Evaluation  Command 
ATTN:  CSTE-ZA 
4501  Ford  Avenue 
Alexandria,  VA  22302 


Commander  Space  &  Naval  Warfare  Systems  Command 
ATTN:  Management  Operations 
National  Center,  Bldg.  1 
Washington,  EXZ  20361 

Commander  Operational  Test  &  Evaluation  Force 
ATTN:  Dest02B 
Norfolk,  VA  23511-6388 


Commander  Air  Force  Operational  Test  & 
Evaluation  Center 
ATTN:  Asst.  Director,  Training 
Building  20130 

Kirtland  Air  Force  Base,  NM  87117-7001 

Commander  Air  Force  Institute  of  Technology 
ATTN:  Student  Operations 
Wright-Patterson  Air  Force  Base,  OH  45433 

Defense  Test  &  Evaluation  Professional 
Institute 

Naval  Air  Warfare  Center,  Weapons  Division 
Code  02PI 

Point  Mugu,  CA  93042 
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